MemT-1.0: Towards Memory-Native Evolution
Zhiyu Li, Zeyu Zhang, Linyun Liu, Yiwei Bu, Chenyang Xi, Xichong Zhang, Yi Yu, Yu Yu, Zhichen Liu, Yuxiao Yi, Chunyu Li, Haoran Liu, Zhenkai Ding, Le Zhou, Juncheng Zhang, Boyu Chen, Zehao Lin, Guorui Li, Renyu Fu, Ji Li, Youyang Xia, Jiajun Shen, Harvey Li, Jingsong Zhou, Helan Hu, Tong Shen, Tianxiao Xu, Chenye Han, Weijie Li, Zhongquan Zhou, Hongming Xu, Yuekeng Li, Erenkai Zhu, Peng Yu, Wenhao Li, Jingyi Kang, Xixuan Hao, Xuzhou Zhu, Can Chen, Jiawei Liu, Chen Jiang, Tao Jiang, Fangwei Xiang, Zepeng Yao, Kai Chen, Guang Yang, Mingfei Lu, Chunhui Li, Zhengxuan Jiang, Bo Tang, Shiyu Zhan, Yihang Sun, Jie Chu, Wenjun Xiong, Zhenxiang Hua, Chen Lin, Detong Su, Dongyu Deng, Qiuping Ding, Guofeng Ding
MemT-1.0 internalizes memory as a native model capability using stable token-level updates and Mixture-of-Storage.
How can we integrate persistent, evolving memory directly into Transformer architectures to enable continuous information accumulation across interaction steps?
Foundation models typically rely on external context windows or retrieval systems to access historical information, which scales poorly and forces the model to re-process raw text for every interaction. MemT-1.0 replaces this external dependency with a native memory architecture that uses Gated DeltaNet-2 to perform stable, token-level memory updates within each model layer. It scales this capacity using Mixture-of-Storage, where dense routers distribute information across parallel memory experts. On memory benchmarks, MemT-1.0 outperforms existing parametric memory baselines while maintaining general capabilities, achieving a 66× reduction in persistent storage compared to full-context KV caching.
Paper Primer
The core mechanism hinges on a parallel memory-attention block that maintains a dynamic state $M_t$ alongside the standard Transformer backbone. The memory update is like a ledger: it reads incoming tokens, decides what to erase or write via gated projections, and updates the state incrementally so that the model carries forward only the distilled essence of past interactions.
MemT-1.0 achieves superior memory retention compared to parametric baselines without requiring full-context textual input.
Across 11 memory benchmarks, MemT-1.0-Flash and MemT-1.0-Nano outperform the strongest parametric memory baselines by 12.5 and 8.9 points, respectively. At a 128K token history length, MemT-1.0-Nano provides a 1.86× inference speedup and a 66× reduction in persistent storage relative to full-context KV caching.
The training pipeline uses an off-policy distillation framework where a frozen, full-context backbone acts as a teacher. The student model is trained to approximate the teacher's output distribution using only the compressed memory state, effectively learning to "read" its own internal history to guide generation.
Why use a native memory architecture instead of just increasing the context window?
Increasing the context window leads to quadratic compute costs and requires re-processing the entire history for every new token. MemT-1.0 maintains a constant-size persistent state, allowing for efficient inference even as interaction history grows indefinitely.
Does internalizing memory degrade the model's general reasoning or coding abilities?
The authors observe that MemT-1.0 largely preserves general capabilities. While some benchmarks show minor fluctuations, the model remains competitive with its backbone on tasks like MMLU-Pro and BigCodeBench-Hard, even when distracting noise is injected into the memory.
MemT-1.0 demonstrates that memory can be treated as a scalable, internal model parameter rather than an external data-retrieval problem, enabling long-term experience accumulation in agentic workflows.
Introduction to MemT-1.0
MemT‑1.0 introduces evolving native memory to replace fixed context windows.
Current foundation models are limited by fixed context windows; they cannot retain information across long interaction steps. MemT‑1.0 addresses this gap by embedding a persistent native memory that evolves with each interaction, enabling continual accumulation of experience. This shift opens a path toward truly memory‑native evolution.
The model maintains a latent state that is updated at every interaction step, so knowledge can be carried forward beyond the static context window.
**Figure 2.** Agentic performance of MemT-1.0-Flash after backbone preparation without native memory.
**Figure 3.** Inference efficiency and storage overhead of MemT-1.0-Nano.
The key shift is moving from a static context window to an evolving native memory that persists across interaction steps.
Model Architecture
Stable Mechanism I details the native memory block that adds parallel memory attention to each Transformer layer.
Fixed‑length context windows force Transformers to forget information older than a few thousand tokens, which hampers tasks that require long‑range consistency.
The block equips each selected Transformer layer with a persistent memory that is read and written at every interaction step, running side‑by‑side with the usual self‑attention path.
How does this memory block differ from the external KV‑cache used in retrieval‑augmented generation?
The KV‑cache stores static key‑value pairs that never change during generation. In contrast, the native memory block is updated every interaction step by learned erase/write/forget gates, so its content evolves with the conversation and can encode dynamic state.
Compute RMSNorm of $\tilde H^{(l)}_{t}$ → $[[0.71,0.71],[0.71,0.71]]$ (each row normalized to unit RMS).
Project $\tilde H^{(l)}_{t}$ through $W^{(l)}_{g/d}$ and $W^{(l)}_{g/u}$ (both identity for this toy example) to obtain $G^{(l)}_{t,h}= \tilde H^{(l)}_{t}$ for each channel $h$.
Apply SiLU: $\text{SiLU}(1)=1\cdot\sigma(1)=0.73$, $\text{SiLU}(0)=0$; thus $G$ becomes $[[0.73,0],[0,0.73]]$.
Apply SoftCapSiLU with $\tau=2$: $\text{SoftCapSiLU}(0.73)=2\tanh(0.73)=1.30$, $\text{SoftCapSiLU}(0)=0$.
Element‑wise multiply RMSNorm and SoftCapSiLU → $[[0.71\times1.30,0.71\times0],[0.71\times0,0.71\times1.30]] = [[0.92,0],[0,0.92]]$.
Concatenate the two channel results (identical here) → final $A^{(l)}_{\text{mem},t}$ is a $2\times2matrix[[0.92,0],[0,0.92]]$.
The capped SiLU keeps the memory‑attention values bounded, preventing runaway activations while still allowing each channel to contribute proportionally to the hidden state.
Native Memory Storage Procedure – token‑wise update using learned gates.
During utilization, the current memory state is queried with a normalized hidden vector, scaled by $\sqrt{d_k}$, and gated by a capped SiLU before being added to the original attention output.
Parallel original and memory attention lets the model keep the fast, dense self‑attention path while gaining a long‑range, stateful read/write channel that does not blow up the quadratic cost.
**Figure 4** Architecture of MemT-1.0. MemT-1.0 equips causal language models with native memory that evolves across interaction steps. Each selected Transformer block is augmented with a native memory block that performs parallel original and memory attention. The memory mechanism comprises dynamic local memory and trainable hyper memory, together with native memory storage and utilization procedures for updating and retrieving persistent information.
Native Memory Blocks give Transformers a persistent, learnable memory that scales linearly with sequence length while preserving the original attention speed.
Training Pipeline
Outlines the two‑stage training pipeline that equips MemT‑1.0 with evolving native memory.
Fixed‑length context windows force Transformers to forget earlier tokens, which hampers tasks that require long‑range reasoning. The pipeline below injects a persistent native‑memory state that carries information across interaction steps, eliminating that bottleneck.
The loss blends a standard cross‑entropy term that teaches the model to predict the next token, with a KL‑divergence term that forces the student’s output distribution to match a full‑context teacher, all under a mask that selects which tokens receive supervision.
Compute the CE loss for each token (e.g., $0.8$ and $0.6$).
Compute the KL loss for each token (e.g., $0.4$ and $0.3$).
Weight each term by $0.5$ and sum: token 1 contributes $0.5\times0.8 + 0.5\times0.4 = 0.6$, token 2 contributes $0.5\times0.6 + 0.5\times0.3 = 0.45$.
Apply the mask (both are $1$), so the total loss is $0.6 + 0.45 = 1.05$.
The example shows how the mask isolates supervised positions while the $\lambda$ weights smoothly interpolate between pure CE and pure KD training.
How does this loss differ from a naïve multi‑task loss that simply adds CE and KD terms for every token?
Because the mask $m(b)_{t,k}$ disables the loss on tokens that are not relevant at the current stage (e.g., history tokens during CPT). Without the mask, the model would be forced to predict or distill on positions that have no ground‑truth target, leading to noisy gradients and destabilizing memory updates.
The first stage, CPT, trains the model to perform causal language modeling within each fixed‑size segment while reading earlier segments exclusively through native memory. This forces the model to rely on the evolving memory state rather than on an ever‑growing textual context.
Split each training sequence into consecutive, non‑overlapping segments of length $L$ (e.g., $L=2\,\text{k}$ tokens).
For the current segment $t$, feed the tokens to the Transformer backbone and compute CE loss on the next‑token targets.
Read the current native‑memory state $M_t$ and incorporate it into the attention mechanism.
After processing the segment, apply the native‑memory update to obtain $M_{t+1}$, which will be used for the next segment.
Repeat until the final (possibly shorter) segment is processed; then start a new independent sequence with $M_0=\mathbf{0}$.
The second stage, post‑training, refines the model for response generation. Here the student receives a query and a partially generated response prefix, while the teacher provides the full‑context distribution. The loss switches to KL divergence, and the mask now selects only the response tokens that are supervised.
Initialize memory parameters from the CPT checkpoint; set $M_0$ to zero for each new trajectory.
For each interaction step $t$, feed the current query $X_t$ and the observed response prefix $Y_{t,<k}$ to the student.
Compute the KL loss between the student’s output distribution and the teacher’s distribution, masked so that only response tokens (not query, history, or padding) contribute.
Back‑propagate through the memory readout and the preceding native‑memory updates, allowing gradients to shape how history is stored.
Update the native‑memory state $M_{t+1}$ with the token‑level write operation before proceeding to the next query.
MemT‑1.0‑Nano follows a single CPT epoch then post‑training. It uses a global batch of 64 packed sequences (max 64 k tokens per pack) and a segment length of 2 k tokens. AdamW optimizes only the memory parameters with $\beta_1\!=\!0.9$, $\beta_2\!=\!0.999$, $\epsilon\!=\!10^{-8}$, weight decay 0.01, and a gradient‑clipping norm of 1.0. Mixed‑precision training keeps the backbone in BF16 while memory‑sensitive components stay in FP32. A soft cap of 2.0 limits the memory output gate. The learning‑rate schedule warms up for 600 steps to $1\times10^{-4}$, holds that peak, then decays cosine‑wise over the final 1 000 steps.
MemT‑1.0‑Flash reuses the same backbone but scales the batch to 192 packs. The peak learning rate is $8\times10^{-5}$ with a 100‑step warmup, followed by cosine decay. All other hyper‑parameters (segment length, optimizer settings, soft‑cap) remain identical to the Nano configuration.
Balancing KD and CE in the loss yields stable memory updates while letting the model learn both immediate token prediction and long‑range teacher alignment.
System and Infrastructure
Describes the stable‑update trick that makes memory updates cheap and scalable.
Native memory updates grow linearly with the number of experts, and each step would otherwise require a full matrix‑vector product and an outer product. This cost quickly dominates GPU memory when the pack length or expert count increases.
The update rewrites the rank‑one addition so that the expensive outer product can be folded into a triangular solve, letting many small updates be batched together without materialising full expert projections.
Compute $D_1 = \operatorname{Diag}([1, 2])$.
Compute $u_1 = \sqrt{1}\,\odot k_1 = [1, 0]$ and $z_1 = 1 \odot v_1 = [0, 1]$.
Scale the previous state: $D_1 S_0 = \begin{bmatrix}1 & 0\\0 & 2\end{bmatrix}$.
Rank‑one term: $u_1^{\top} z_1 = [1,0]^{\top}[0,1] = 0$ (scalar zero).
Interaction subtraction: $u_1 D_1 S_0 = [1,0]\begin{bmatrix}1 & 0\\0 & 2\end{bmatrix} = [1,0]$.
Combine: $S_1 = D_1 S_0 + 0 - [1,0] = \begin{bmatrix}0 & 0\\0 & 2\end{bmatrix}$.
The update eliminates the need to materialise a full $2\times2$ outer product; only a diagonal scaling and a few vector ops are required.
Windowed execution groups complete steps into “windows” so that expert projections are performed only for the tokens inside the current window. Smaller windows shrink the live intermediate tensors, but each window incurs an extra launch and a recomputation of the backward pass.
In this configuration, processing two steps at a time reduces peak memory by approximately 80%, with a 1.6% increase in median F+B time. Shrinking the window to one step saves additional memory but raises the time.
How does the stable update differ from a naïve additive memory write?
In a naïve additive write the new state would be $S_i = S_{i-1} + k_i^{\top} v_i$, which forces a full outer‑product per channel. The stable update rewrites this as a diagonal scaling plus a rank‑one term and an interaction subtraction, allowing the outer‑product to be absorbed into a cheap triangular solve and batched across many chunks.
Evaluation Results
Evaluation quantifies memory performance, inference speed, storage cost, and low‑rank compression of MemT‑1.0‑Nano.
MemT‑1.0 integrates native, evolving memory into Transformer blocks, letting the model keep a persistent latent state that updates across interaction steps. This section measures how that design translates into concrete performance gains.
Update granularity (token‑level vs. turn‑level) dominates architecture impact, dropping the average score from 46.97 to 40.98 when switching to turn‑level aggregation.
Table 4 shows the “turn‑level” variant reduces the average by $5.99$ points compared to the baseline token‑level update.
**Figure 1.** Memory performance of MemT-1.0 on representative benchmarks.
MemT‑1.0‑Nano achieves up to $3.99\times$ end‑to‑end speedup over full‑context inference at a 256 K‑token history.
Figure 8 reports the latency curves; the 256 K point shows a $3.99\times$ advantage.
MemT‑1.0‑Nano reduces per‑trajectory storage by $66\times$ at 128 K tokens compared with full‑context inference.
Figure 9 shows the storage curve for MemT‑1.0‑Nano remains flat while full context grows linearly, yielding a $66\times$ reduction at 128 K tokens.
Low‑rank decomposition improves accuracy as rank $k$ grows, but returns diminish quickly; on PersonaMem the best accuracy appears around $k=64$ and plateaus thereafter.
Figure 10 shows accuracy rising with $k$ up to an intermediate point, then flattening despite larger storage.
Scaling Native Memory
Introduces Memory of Experts, a scaling technique that expands native memory into multiple dense experts.
Native memory blocks are limited in capacity, which becomes a bottleneck when agents need to store many procedural experiences. The trick that resolves this bottleneck is to split each block into several “experts” that each hold a portion of the data, while a pair of dense routers decide where to write and read. This design lets the model keep the same overall memory footprint but increase the amount of information it can retain.
Think of a library that, instead of a single shelf, has several parallel shelves (experts). A librarian (the Utilization Router) looks at the book’s label and drops it onto the appropriate shelf; later, a patron (the Storage Router) asks for the book and the librarian fetches it from the right shelf. The shelves share the same address space, so the library can hold more books without expanding the building.
Write phase: the token $x$ is split—70 % of its information is stored in expert 1’s slots M₁, 30 % in expert 2’s slots M₂.
After the write, expert 1’s storage matrix becomes M₁ ← M₁ + 0.7·x, and expert 2’s becomes M₂ ← M₂ + 0.3·x.
Read phase: when the model later attends, the Storage Router applies $\alpha$ₛ, retrieving 0.4·M₁ + 0.6·M₂ as the memory contribution.
The combined memory vector is then added to the token’s hidden state, influencing the next transformer layer.
Even though the total number of slots stays at 4, the two experts can store partially overlapping information, effectively increasing the diversity of representations the model can recall.
How does MoS differ from a standard Mixture‑of‑Experts (MoE) layer?
In a conventional MoE, each token is routed to a single expert (sparse routing) and only that expert’s parameters are updated. MoS uses dense routing: every token contributes to all experts via learned weights, and the experts share a common address space. This means MoS retains the full capacity of the original memory block while still gaining the representational benefits of multiple experts, whereas MoE reduces the effective parameter usage per token.
**Figure 5** Scaling native memory capacity. MoS expands each native memory block into multiple memory experts and employs separate dense storage and utilization routing over shared expert addresses.
Agentic Task Framework
Details of the Experience Bank, trajectory construction, and training pipeline for agentic tasks.
Fixed‑length context windows prevent agents from remembering earlier steps, causing repeated failures when a task requires long‑range reasoning. The core trick is to give the model a persistent latent state that survives across interaction steps, so earlier decisions can be consulted later. This section details the mechanism that makes that possible.
The Experience Bank is a curated notebook of past successful interactions: each entry records the situation, the action taken, and the outcome, enabling the model to retrieve concrete procedural knowledge when faced with a similar state.
Episode A: state signature = {tag: “python”, first‑tool: “pip”, error: “ImportError”}; policy induced → “when ImportError, reinstall package, verify import, watch out for version mismatch”.
Episode B: same signature as A, but the trajectory shows a successful reinstall; it is pooled with A.
Clustering merges A and B into one representative experience containing the policy above and the concrete commands used.
The bank’s specificity means that when the model encounters an ImportError, it can retrieve the exact reinstall sequence rather than a generic “try fixing the environment”.
Generate original agent trajectories on the target task.
Identify up to eight candidate breakpoints per rollout by scanning observations for failures, empty outputs, or repeated errors.
Retrieve the top‑10 experiences from the Experience Bank; form ten candidate subsets (top‑3, top‑5, and eight random three‑entry subsets).
For each retained breakpoint $b$ and experience subset $j$, concatenate the prefix $\xi_{\le b}$, the experience messages $g_{b,j}$, and the continuation $\hat{\xi}(g_{b,j})$ to obtain $\xi_{b,j}$.
Keep $\xi_{b,j}$ as a training sample only if the original trajectory fails ($\text{Success}(\xi)=0$) and the conditioned continuation succeeds ($\text{Success}(\xi_{b,j})=1$).
Training uses hyper‑memory blocks at ten decoder layers; each block holds a fixed $512\times512$ value matrix and a $512\times1$ key normalizer, so memory size is independent of trajectory length. The write side applies top‑$p$ selection with $p=0.9$ and a delta‑update ratio of $0.9$, while the read side fuses memory output with a coefficient $\gamma=0.9$. Two supervision regimes are explored: Full‑Suffix (loss on every assistant turn) and First‑Step (loss only on the first turn after the breakpoint). Both cross‑entropy (CE) and knowledge‑distillation (KD) objectives are evaluated, with KD ultimately chosen for downstream agent performance.
**Figure 11** Experience-guided agentic framework with MemT-1.0-Nano. Retrieved experience is stored in textual context or native memory to guide subsequent actions.
**Figure 13** Construction of experience-conditioned training trajectories. Retrieved experience is added at selected breakpoints in unsuccessful trajectories, and execution resumes from the same state. Continuations that pass the task tests are retained as training samples comprising the trajectory prefix, experience, and successful continuation.
**Figure 14.** Training loss and validation NLL under CE and KD objectives. Within each panel, the left and right columns show training loss and validation NLL, respectively. Solid blue curves denote Max-2, and dashed orange curves denote Max-3.
**Figure 15.** Agentic coding performance on SWE-bench Verified and SWE-bench Multilingual under Before Execution (top) and Harness-Triggered (bottom). Each task is evaluated in three runs.
Memory Windowing Experiment
Ablation shows windowed execution cuts memory dramatically while adding modest latency.
We compare two execution schedules for a single MoS memory block: whole‑pack execution that processes all tokens at once, and windowed execution that processes tokens in sequential windows, recomputing intermediates as needed.
Instead of a single monolithic pass, the model splits a pack into smaller windows, runs each window’s projections, updates the latent state, and reads out results before moving to the next window.
How does windowed execution differ from a naïve sliding‑window inference that simply overlaps windows?
Sliding‑window inference reads overlapping chunks but never updates the persistent memory state; windowed execution updates the latent state after each window, allowing information to accumulate across windows.
**Table 14.** Configuration of the MoS memory-block experiment.
As pack length increases from 8K to 32K, whole-pack peak memory grows from 12.37 GiB to 47.83 GiB, while 4096-token windows grow from 8.36 GiB to 9.49 GiB. Windowing bounds the lifetime of projections and intermediates, while retained outputs and boundary states still grow with length. Across the three lengths, the median time increases for 4096-token windows are 15.0%, 11.1%, and 1.6%, respectively, showing that windowing overhead depends on the workload.
All samples are retained, including whole-pack trials at 16K/2048 (378.03 ms) and 32K/2048 (562.40 ms), and a windowed trial at 32K/4096 (640.58 ms). The 1.6% reported in the chapter therefore describes the median time difference; individual trials exhibit variability.
Qualitative Case Studies
Four conversation examples illustrate MemT-1.0‑Nano’s memory behavior up to 64 K tokens.
MemT-1.0‑Nano maintains correct memory behavior across conversation histories up to 64 K tokens.
Four case studies at 4 K, 16 K, 32 K, and 64 K tokens each show the model answering according to the stored latent memory slot.
The four examples progress from simple fact retrieval to handling missing data, rule chaining, and update tracking, illustrating how the single latent memory slot evolves as the conversation grows.
Evaluation Setup
Evaluation setup for memory and general benchmarks, baselines, prompts, decoding, and storage overhead.
We evaluate memory capabilities on eleven benchmarks grouped by trajectory length, covering long‑term retention, dynamic updating, and personalized understanding. Each benchmark is run with independent scopes that preserve the original message order, and no synthetic dates are added when timestamps are missing.
General capabilities are measured on six benchmarks spanning knowledge reasoning, instruction following, long‑context understanding, and code generation. All benchmarks use the same fixed demonstrations for the noise condition, and evaluations follow the official splits and scoring protocols.
Baseline comparisons include a full‑context reference, a dense retrieval system (DenseRAG), and a suite of parametric memory methods such as MemOS, Mem0, OpenViking, Graphiti, Temp‑LoRA, Doc‑to‑LoRA, SHINE, $\delta$‑mem, LiveMem, MemoryLLM, and ICAE. All baselines share the same backbone, prompts, generation settings, and scoring pipelines.
For memory benchmarks, query prompts are constructed without generic few‑shot demonstrations. Each benchmark uses a handcrafted template that inserts the full history (or its truncated suffix) and the current question into a single user message, preserving original roles and timestamps where applicable.
Decoding uses temperature 0.0 for all tasks except the priming subtask of ImplicitMemBench (temperature 0.8). Generation length limits range from 16 tokens (MemBench) to 4 k tokens (ImplicitMemBench), with longer limits for complex tasks such as BEAM summarization.
General‑capability evaluations employ Thinking mode for DeepSeek‑V4‑Flash (high reasoning effort, temperature 1.0) and Qwen3.6‑35B‑A3B (temperature 1.0, `top_p` 0.95). Output budgets are 131 k tokens for DeepSeek and 81 k for Qwen, with YaRN expansion for LongBench v2.
Storage overhead is measured by reporting the logical state size after history preparation, excluding model weights. Full‑context scales linearly with visible tokens, DenseRAG adds FP32 vectors, Temp‑LoRA stores a BF16 adapter, and MemT‑1.0 keeps a full‑rank memory state.