Remory: Learning Residual Memory for Context Compaction

Hanchen Xia, Baoyou Chen, Yutang Ge, Naihao Deng, Senqiao Yang, Zilong Dong, Weihao Yuan, Siyu Zhu

REMORY supplements textual summaries with learned soft tokens to maintain context for long-horizon agents.

How can we supplement lossy textual summaries of long-horizon agent history with learned "residual" memory to recover information lost during context compaction?

Long-horizon agents must periodically compact their history into a summary to fit within finite context windows, but these summaries often omit critical details needed for subsequent decisions. REMORY addresses this by training a neural network to generate a sequence of soft memory tokens that are appended to the summary, acting as a residual connection that provides the frozen actor with missing context. This approach improves agent performance across benchmarks, significantly reducing tool errors and repeated outputs while using only 5.2% of the original context length.

Paper Primer

The core mechanism treats the summary as a base checkpoint and uses a memory network to predict a "residual" sequence of continuous vectors. This sequence is conditioned on the summary and appended to it, allowing the frozen Large Language Model (LLM) to approximate the continuation it would have produced if it had access to the full, uncompressed history.

The training process follows a two-stage curriculum: first, the network learns to reconstruct source passages from soft tokens; second, it learns to minimize the KL divergence between the actor's predictions given the full history versus the actor's predictions given only the summary and the learned memory tokens.

Residual memory improves source attribution and agent task success rates.

On the SummHay benchmark, citation F1 increased by 4.04 points, and across four agent benchmarks, Qwen3.8-27B and GLM-5.3-Flash showed consistent score gains. Agent tool errors were reduced by 25.3% to 49.7% across tested benchmarks.

Why use a learned memory supplement instead of just making the textual summary more detailed?

Textual summaries are limited by the agent's ability to parse and reason over dense prose. Residual memory provides continuous, soft representations that the frozen actor can process as a direct extension of its input, capturing information that may be difficult to articulate or prioritize in a standard text summary.

Does this method replace the need for traditional file retrieval or archival tools?

No; the authors explicitly state that exact records remain accessible through ordinary archive and file-retrieval tools. REMORY is designed to supplement the summary for immediate decision-making, not to serve as a complete, lossless database of all past interactions.

For researchers building long-horizon agents, REMORY demonstrates that "residual" continuous tokens can effectively bridge the information gap left by textual compaction without requiring fine-tuning of the underlying LLM.

The Context Compaction Problem

We expose the tension between limited context length and loss of detail in long‑horizon agents.

Long‑horizon agents operate under a hard limit on the number of tokens they can attend to. As interactions accumulate, the agents must compress the growing history into a fixed‑size textual summary, but this compression can discard the fine‑grained details that later decisions rely on.

Compressing a long interaction into a short summary frees space in the model’s context window, but the summary inevitably loses some of the original details.

**Figure 2.** Context compaction and human memory. (a) Input length across successive requests in an agent run, showing repeated growth and sharp reductions. (b) Memory systems illustrated using the Harvard–Oxford atlas (Harvard–Oxford Atlas Contributors, 2026): prefrontal regions (blue), hippocampus (cyan), and association cortex (purple).

The fundamental trade‑off between context length and information density drives the need for residual memory.

The REMORY Architecture

Introducing Residual Memory, a learned token supplement that preserves information across context compaction.

To keep long‑range agents from losing detail, we augment the compact summary with a learned residual that can be read later.

After the model produces a short textual summary, a small set of learned soft tokens is appended, acting like a “memory tail” that carries the fine‑grained information the summary omitted.

Form $H_i$ by concatenating $S_i$ (2 tokens) and the existing residual (2 tokens) → $H_i$ has 4 tokens.

Apply the REMORY network C_$\phi$ to $H_i$ and $S_i$, producing a new residual $M_i$ of 1 token (since the budget m=5 allows only one more token).

Concatenate the new summary and residual → [$S_i$; $M_i$] now contains 3 tokens, ready for the frozen actor.

The residual token fills the gap left by the short summary, and the budget m guarantees the total length stays bounded regardless of how many boundaries are processed.

For Qwen 3.8‑27B the REMORY network adds 1.94 B trainable parameters, while for GLM‑5.3‑Flash it adds 1.502 B, both producing continuous memory rather than discrete tokens.

**Figure 3.** Residual compensation and training. (a) Codex CLI passes the `/compact` summary to the next actor call. (b) The REMORY network uses that summary to condition memory construction from the active context; summary and continuous memory jointly support subsequent actions. Archive access remains an ordinary tool action in either setting. (c,d) Qwen3.8-27B teacher–student top-1 agreement during Stage I reconstruction and Stage II residual training. The residual panel compares reconstruction initialization with random initialization under the same residual objective and data order. Curves are optimization diagnostics from the training streams.

Training and Benchmarking

Training Procedure shows how residual memory boosts SummHay performance.

The training pipeline combines two ingredients: Codex CLI‑driven context compaction and residual compensation via the REMORY network. Together they enable the agent to retain fine‑grained details that summaries discard.

Codex CLI is a command‑line tool that compresses a long context into a shorter representation while preserving the information needed for downstream tasks.

SummHay measures how well a system recovers relevant insights and correctly cites their sources after the context has been compacted.

Adding residual memory to the summary representation raises Citation F1 by +4.04 points over the Summary++ baseline.

Table 1 shows Citation F1 = 61.03 for “Summary + MEMORY” versus 56.99 for “Summary++”.

Stage I trains the student to reconstruct the original passage from its soft memory by matching the teacher’s predictions.

Stage II extends this by conditioning continuation on a fixed summary; the memory encoder receives only the summary‑derived context, not the future response.

Residual memory delivers a clear citation‑F1 boost on SummHay without sacrificing coverage.

Agent Performance Benchmarks

Benchmark results show Residual Memory consistently raises scores while lowering costs.

The central premise—that long‑horizon agents need a learned supplement to recover details lost in textual summaries—is revisited here. We now examine how Residual Memory performs across several benchmarks.

BrowseComp measures an agent’s ability to retrieve relevant information while navigating a large corpus, emphasizing both relevance and efficiency.

Terminal‑Bench evaluates end‑to‑end task completion on a suite of terminal‑style interactions, reporting success score, monetary cost, and operational robustness.

Residual Memory boosts benchmark scores by up to 6.2 points.

The bootstrap interval for the joint‑score gain is [+0.74, +6.21] across AutomationBench, JobBench, Terminal‑Bench 2.1, and BrowseComp.

**Figure 1.** BrowseComp and Terminal-Bench 2.1 scores. Pale and solid bars compare the same local actor without and with residual memory. Gray bars are published frontier references, with their original harnesses and protocols (Google DeepMind, 2026; Terminal-Bench Team, 2026; OpenAI, 2026b); they are not controlled comparisons with our runs.

System Efficiency and Costs

Residual slots improve task scores while trimming repeats and errors.

We evaluate four research tasks—BrowseComp, Terminal‑Bench 2.1, AutomationBench, and JobBench—using the same actor configuration. All within‑actor factors (weights, Codex CLI, prompts, tools, decoding, budgets, and compaction policy) are held constant, and both configurations can retrieve the exact history.

Qwen operates with a $262{,}144$‑position window, GLM with a $1{,}000{,}000$‑position window, and the intervention adds up to $4{,}096$ residual slots at compaction. Using the full task sets—1,266 BrowseComp, 89 Terminal‑Bench, 600 AutomationBench, and 65 JobBench tasks—we run each actor once except Qwen on AutomationBench (five runs).

Qwen improves by $9.8$ points on AutomationBench and $7.6$ points on JobBench, extending gains to business workflows. It also gains $4.5$ points on Terminal‑Bench and $3.0$ points on BrowseComp. GLM shows $3.3$ points and $4.1$ points improvements on the same two benchmarks, confirming the effect holds for a stronger actor with a larger context.

**Table 3.** Terminal-Bench 2.1 and BrowseComp: one run per configuration. Costs are mean API-equivalent USD per task. Repeats and errors are tool-output counts over the full task set.

Qualitative Case Studies

Real‑world case studies reveal how Residual Memory changes trading returns and visual‑task repairs.

Live Trading compares two concurrent Qwen paper‑trading accounts that follow the same context policy, differing only by the presence of Residual Memory.

**Figure 4.** Live Trading context use. (a) An illustrative September 22 trace. (b) Compaction intervals in the separate September 20–23 evaluation: 56/45 completed compactions and 51/41 within-session intervals without/with residual memory. Intervals do not cross restarts; two residual intervals exceed the display range but remain in the box statistics.

At the September 23 cutoff the account without Residual Memory ends with a –11.86 % ROI on a 20,000 $ bankroll, while the account with Residual Memory achieves +5.11 % ROI, and the generation cost drops from 303.81 $ to 170.30 $.

Compactions decrease from 56 to 45 as total responses fall from 5,336 to 3,013, yet per 1,000 responses compactions rise from 10.5 to 14.9, reflecting fewer interactions; mean live context shrinks from 105,372 to 101,185 positions, explaining the lower generation spend.

**Figure 5.** Cell segmentation after repair. The task image and the exact overlay viewed by the actor are reproduced without pixel changes.

In the Terminal‑Bench cell‑segmentation task a refactor removed helper functions before compaction; with summary and Residual Memory the actor rereads both files, restores the helpers, and passes all nine verifier tests, illustrating how summary, memory, and retrieval complement each other.

These case studies show that Residual Memory can boost real‑world performance, delivering higher returns in live trading and reliable repairs in visual‑task pipelines.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers