Continual Learning Mechanisms Compose for Long-Horizon Memorization
Zheyuan Zhang, Alvin Zhang, Daniel Khashabi, Tianmin Shu
Composing data, function, and weight anchors with merged LoRA enables language models to retain associations across 100 sequential tasks.
How can we combine existing continual learning mechanisms to enable language models to reliably memorize information across 100 sequential tasks?
Language models suffer from catastrophic forgetting when fine-tuned on a long sequence of tasks, losing earlier knowledge as they learn new information. The authors hypothesize that combining complementary mechanisms—data replay, self-distillation, and weight regularization—can preserve memory more effectively than any single approach. They organize these into a design space of "anchors" and "low-rank allocation rules" to systematically test compositions. Combining all three anchors with merged LoRA achieves a 28-fold improvement in final retention compared to naive sequential fine-tuning, ranking among the top methods across all tested datasets.
Paper Primer
The paper defines "long-horizon memorization" as the challenge of retaining associations across 100 sequential query-answer tasks without task identifiers. Because the model cannot revisit raw training data from earlier tasks, updates for new tasks inevitably overwrite previously stored knowledge.
The method composes three "anchors" to constrain updates: data anchors (generative replay), function anchors (self-distillation), and weight anchors (importance-based regularization). These are paired with merged LoRA: folding low-rank updates into the dense weights after each task to keep the model's state size constant while accumulating knowledge.
Composing complementary mechanisms substantially outperforms standalone continual learning techniques.
The best composition (all three anchors + merged LoRA) achieved 34.9% average final retention across three 100-task datasets, compared to 1.2% for naive fine-tuning and 8.1% for the best individual mechanism.
Data replay and merged LoRA are the primary drivers of retention.
Factorial analysis shows these two mechanisms provide the largest main effects and interact super-additively on all datasets.
Why is this problem framed as "memorization" rather than "generalization"?
The authors focus on the model's ability to recall specific query-answer associations seen during training. They explicitly note that their results do not establish generalization to paraphrased queries or new formulations.
Does this approach preserve the model's general capabilities?
No. The authors report that even with improved memorization, the models still exhibit catastrophic forgetting on general benchmarks like MMLU and GSM8K after 100 tasks.
The Long-Horizon Memorization Setting
We expose why single mechanisms fail and how composing anchors with low‑rank updates enables long‑horizon memorization.
Language models quickly forget earlier tasks when trained sequentially, limiting their ability to retain knowledge over long horizons. Single‑mechanism continual‑learning tricks (e.g., replay or regularization) each address only one failure mode, leaving a gap for robust long‑horizon memorization.
Retaining accurate behavior across dozens or hundreds of sequential tasks requires preventing forgetting while still learning new data.
**Figure 1:** Memory lifetime under the best single mechanism and composed continual learning methods. Lines show three-seed means with min-max bands. Half-life counts tasks until retention halves. Combining multiple anchors with merged LoRA substantially reduces catastrophic forgetting and extends memory lifetime beyond the best single mechanism.
The key insight is that moving from a single‑task setting to memorizing 100 sequential tasks requires composing complementary anchors with a low‑rank allocation strategy.
Evaluation Protocol
Evaluation setup defines metrics, datasets, and a successive‑halving search for efficient method selection.
Task‑level successive halving reduces the candidate pool from 90 to 10 configurations after 50 tasks.
Starting with 90 configurations, TSH keeps the top 10 after evaluating retention at task horizon 50.
We follow the domain‑incremental protocol of Van de Ven & Tolias (2019): the model receives one task at a time and is not given the task identity during inference. Each task consists of query‑answer pairs, and because we study memorization rather than generalization, evaluation uses the same examples as training.
We evaluate three memorization datasets that increase in semantic realism. Symbol‑QA contains 10,000 random key‑value pairs, LLM‑QA contains 10,000 LLM‑generated query‑answer pairs across 100 fictional topics, and Real‑QA contains 5,000 natural QA pairs filtered to exclude items the model answers correctly in any of five sampled completions.
Crossing the three anchor categories with low‑rank allocation rules yields 90 candidate continual‑learning methods. We apply Task‑level Successive Halving (TSH) to rank these configurations by retention over progressively longer task horizons, retaining the top 45 after 10 tasks, top 23 after 20 tasks, and top 10 after 50 tasks, which then train through all 100 tasks.
Experimental Findings
Best composition (all three anchors + merged LoRA) attains 34.9 % average final retention across three datasets.
Figure 3 visualizes final retention for every anchor‑plus‑merged LoRA combination, revealing a clear hierarchy of performance.
The composition that includes all three anchors together with merged LoRA is the only method that ranks among the top three across all three datasets.
Figure 3 shows this composition achieving 23.2 % on Symbol‑QA, 41.8 % on LLM‑QA, and 54.8 % on Real‑QA, which averages to 34.9 %.
**Figure 3.** Final retention (%) after 100 tasks for all 16 combinations, averaged over three seeds. Filled markers identify active mechanisms, and an inactive Merge marker indicates shared LoRA. The leftmost column is shared LoRA without anchors. Bold marks the best result per dataset. Appendix E.2 reports standard deviations over seeds. The best compositions substantially outperform standalone mechanisms.
**Figure 4.** Greedy buildup paths through the $2^4$ factorial. Each step adds the remaining mechanism with the highest final retention under the current composition. Triangles mark the strongest composition along each path.
**Figure 5.** Temporal accuracy matrices $M_{i,j}$ along the composition chain, one row per dataset. Each column adds one mechanism and never removes one. The panel outlined in red is the strongest configuration in that row. Panels show the median seed of three, so no panel is a favorable draw.
**Figure 6.** General capability after 100 tasks, averaged over three seeds. Anchors and hyperparameters from TSH are fixed across allocation rules: all three anchors on SYMBOL-QA, and data and weight anchors on LLM-QA and REAL-QA. O-LoRA better preserves general capability on the two natural-language datasets.
Composition of anchors is necessary for 100‑task retention.
Mechanism Taxonomy
How anchor mechanisms and low‑rank allocation together enable long‑horizon memorization.
Long‑sequence language models quickly forget earlier tasks, so a single trick is needed to keep past information alive without exploding memory.
An anchor is any component that preserves a snapshot of earlier‑task knowledge so the model can refer back to it while learning new tasks.
Attempt 1: sample $\mathbf{z}^{(1)} = (s,\text{"Q1"},\text{"A1"})$ (length 3).
Attempt 2: sample $\mathbf{z}^{(2)} = (s,\text{"Q2"})$ (length 2); the generation stops early, so we keep the two‑token continuation.
Form the replay set $D^{\text{replay}}_2 = \{(s,\mathbf{z}^{(1)}), (s,\mathbf{z}^{(2)})\}$.
During each minibatch we pair the current data with one replay example, compute the blended loss $(1-w)L^{\text{SFT}}_2 + wL^{\text{D}}_2$, and take a gradient step.
Even with only two short generated sequences, the replay loss forces the student to honour the previous model’s token‑level predictions, keeping earlier knowledge alive without storing any raw examples.
Low‑rank allocation decides how a small, trainable LoRA adapter is attached to each layer and how those adapters are carried across tasks.
**Table 2.** Methods used in our project, categorized as anchors or low-rank allocation rules. “Constant” refers to a method that depends only on a fixed set of information, while “Linear in tasks” means the method needs to store information that grows with the number of tasks during training.
**Algorithm 1:** Unconditional generative replay for task $t$. The previous model generates the replay set before the student receives its first update on task $t$. **Input:** task data $\mathcal{D}_t$; student $\Theta$; previous model $\bar{\Theta}_{t-1}$ when $t > 1$; seed $s$; attempts $N_R$; generation temperature $\tau_G$; maximum length $L_{\text{max}}$; replay temperature $\tau_D$; replay weight $w$; epochs $E$ **Output:** updated student parameters 1. **if** $t = 1$ **then** 2. $\quad$ train on $\mathcal{L}_{\text{SFT}}^1$ $\quad$ // $a_D^1 = 0$ 3. **else** 4. $\quad$ freeze $\bar{\Theta}_{t-1}$ 5. $\quad$ $\tilde{\mathcal{D}}_t \leftarrow \emptyset$ 6. $\quad$ **for** $m \leftarrow 1$ **to** $N_R$: 7. $\quad \quadsample\tilde{z}_m \sim p_{\bar{\Theta}_{t-1}}(\cdot \mid s)$ with temperature $\tau_G$, top-$p = 0.9$, and limit $L_{\text{max}}$ 8. $\quad \quad$ **if** $\tilde{z}_m \neq \emptyset$: 9. $\quad \quad \quad$ add $(s, \tilde{z}_m)$ to $\tilde{\mathcal{D}}_t$ 10. $\quad$ **for** $e \leftarrow 1$ **to** $E$: 11. $\quad \quad$ reshuffle the replay iterator 12. $\quad \quad$ **forall** current minibatches $\mathcal{B} \subset \mathcal{D}_t$ **do** 13. $\quad \quad \quad$ draw replay minibatch $\tilde{\mathcal{B}} \subset \tilde{\mathcal{D}}_t$ $\quad$ // recycle the iterator if needed 14. $\quad \quad \quad$ $\mathcal{L} \leftarrow (1 - w)\mathcal{L}_{\text{SFT}}^t(\Theta, \mathcal{B}) + w\mathcal{L}_D^t(\Theta, \tilde{\mathcal{B}})$ 15. $\quad \quad \quad$ add the active function, weight, and low-rank allocation terms from Equation 14 16. $\quad \quad \quad$ take one optimizer step 17. $\quad$ discard $\tilde{\mathcal{D}}_t$ and release $\bar{\Theta}_{t-1}$
Function Anchors
We introduce self‑distillation and weight‑anchor penalties to preserve knowledge across tasks.
Long‑sequence language models drift as new tasks arrive, causing previously learned knowledge to fade. To curb this drift we add two complementary anchors that constrain the model’s behavior at each task boundary.
At the start of a new task we freeze the model from the previous task and force the current model’s token‑wise output distribution to stay close to that frozen teacher using a forward‑KL penalty.
Compute teacher softmax: $\bar{q}= \text{softmax}(2,0,-1,-2) \approx (0.71, 0.26, 0.03, 0.01)$.
Compute student softmax: $p= \text{softmax}(1,0.5,-0.5,-1) \approx (0.44, 0.30, 0.15, 0.11)$.
Calculate forward KL for this position: $\operatorname{KL}(\bar{q}\,\|\,p)=\sum_i \bar{q}_i\log(\bar{q}_i/p_i) \approx 0.71\log\frac{0.71}{0.44}+0.26\log\frac{0.26}{0.30}+0.03\log\frac{0.03}{0.15}+0.01\log\frac{0.01}{0.11}\approx 0.38$.
Since there is only one non‑padding token, $L^{F}_{t}= \tau_F^{2}\times 0.38 = 0.38$.
Forward KL forces the student to keep the teacher’s high‑confidence predictions, so any large deviation (e.g., swapping the top‑probability token) incurs a steep penalty.
Why does the method use forward KL instead of the more common reverse KL for distillation?
Forward KL penalizes the student for assigning low probability to tokens the teacher deems likely, which directly protects the teacher’s confident predictions. Reverse KL would instead punish the teacher for assigning low probability to the student’s predictions, which is less useful when the teacher is frozen and we want to keep its knowledge intact.
After each task we estimate how important each LoRA scalar is for the knowledge just acquired, then add a quadratic penalty that resists changes to those important scalars in future tasks.
Current values before task $t+1$: $\vartheta_1=0.6$, $\vartheta_2=-0.25$.
Compute deviations: $\Delta_1 = 0.6-0.5 = 0.1$, $\Delta_2 = -0.25 - (-0.3) = 0.05$.
Apply quadratic penalty: $R^{\text{SI}}_{t+1}= 2.0 \times (0.1)^2 + 0.5 \times (0.05)^2 = 2.0 \times 0.01 + 0.5 \times 0.0025 = 0.02 + 0.00125 = 0.02125$.
The penalty is larger for the more important coordinate ($\vartheta_1$), discouraging it from moving far from its reference while allowing the less‑important coordinate more flexibility.
How does this weight‑anchor penalty differ from classic EWC applied to full model parameters?
Classic EWC treats every parameter independently and stores a Fisher for each, which grows with model size. Here we restrict the penalty to the small set of LoRA scalars ($P\ll$ total parameters) and reuse the same importance estimate across low‑rank updates, making the memory footprint constant and the penalty focused on the most plastic parts of the model.
Weight‑anchor update at task $t$ (Algorithm 2)
Together, the function anchor keeps the model’s output distribution stable, while the weight anchor protects the most important LoRA parameters, enabling reliable long‑horizon memorization across many tasks.
Context and Prior Approaches
Survey of continual learning approaches and their relevance to long‑horizon memorization.
When a model is trained on new tasks, updates can overwrite knowledge from earlier tasks, causing the model to lose previously learned information.
Prior work on continual learning distinguishes task‑, domain‑, and class‑incremental scenarios, each differing in whether task identity is known at inference and how the output space evolves.
Existing methods fall into three families: regularization (e.g., weight regularization that limits parameter drift), replay (using stored or generated examples to rehearse past tasks), and parameter isolation (allocating separate sub‑networks per task). Hybrid approaches such as Dark Experience Replay combine replay with knowledge distillation, while Momentum Knowledge Distillation adds a teacher constraint to online continual‑learning pipelines.
Early benchmarks focused on sequential image classification (MNIST, CIFAR, ImageNet), but recent extensions target language modeling, instruction tuning, and multitask learning. Low‑Rank Adaptation (LoRA) and its variants (ReLoRA, O‑LoRA, OSRM) aim to preserve prior knowledge by injecting trainable low‑rank updates, while sequential model editing studies the accumulation of targeted edits and their interference.
AgentOdyssey evaluates continual learning agents in text‑based environments, highlighting challenges of world‑knowledge acquisition and episodic memory, which parallel the long‑horizon memorization problem addressed here.
Design Space Formalization
We situate our work among prior continual‑learning approaches and low‑rank adaptation methods.
Continual learning trains a single model on a sequence of tasks while preserving performance on earlier data. Standard formulations distinguish task‑, class‑, and domain‑incremental learning based on output‑space changes and whether inference receives a task identifier. Our setting uses a uniform question‑answer interface and no task identifier at evaluation, testing whether one model can retain all associations without routing inputs to task‑specific predictors.
Regularization methods constrain learning by retaining information from previous tasks. Weight‑based methods assign importance to individual parameters and penalize changes to important values (e.g., EWC, online EWC, SI, MAS). Function‑based methods instead constrain model behavior, exemplified by Learning without Forgetting, which asks the current model to match a previous model’s outputs on available inputs.
Replay methods train on earlier examples stored in memory or on samples generated to approximate earlier data. Parameter‑isolation methods reduce interference by assigning different parameters to different tasks or by restricting which parameters each task may change. Our data, function, and weight anchors instantiate the first three preservation signals, while low‑rank allocation rules determine whether successive task updates reuse or separate low‑rank capacity.
Prior work combines preservation signals, but it does not systematically study the broader space of mechanism compositions considered here. Dark Experience Replay stores examples together with the logits produced when those examples entered memory, then uses both rehearsal and output matching during later learning. Momentum Knowledge Distillation adds a slowly updated teacher to online continual learning methods and studies how distillation complements replay.
Prior language work studies sequential language modeling, task learning, and instruction tuning. LAMOL learns to answer current‑task examples and generate pseudo‑examples for earlier tasks. Continual pretraining instead updates a language model as new corpora, domains, or time periods arrive, but these studies primarily measure language modeling, transfer, or downstream task performance.
We isolate associative retention by evaluating the same question–answer items before and after many later tasks, focusing on recall rather than generalization to new query formulations.
REPINA mitigates representation collapse during fine‑tuning by matching fine‑tuned representations to pretrained representations, either directly or through a learned projection. Our function anchor instead matches the previous model’s output distributions on current‑task data.
Sequential model editing also asks a model to retain many updates. GRACE leaves the original model weights unchanged and stores edits in a discrete key‑value codebook that activates for inputs near a stored key. Repeated application of model editors can weaken earlier edits, reduce the model’s ability to learn new edits, and harm downstream performance.
LoRA represents an update to a frozen weight matrix with two low‑rank matrices that can be merged into the dense weight for evaluation. ReLoRA repeatedly merges and reinitializes low‑rank matrices during pretraining, allowing multiple low‑rank updates to produce a higher‑rank cumulative change.
Various continual‑learning methods exploit this structure: CoLoR trains a separate LoRA expert for each task; O‑LoRA retains the matrices learned for earlier tasks and penalizes overlap between the current and earlier A matrices; InfLoRA constructs task‑specific update subspaces to reduce interference; OSRM uses task features to initialize LoRA subspaces before independently training and later merging task models. Our comparison separates the low‑rank allocation rule from the preservation objective, distinguishing Shared LoRA (reuses one pair), merged LoRA (commits each task’s update to the dense weights before creating a new pair), O‑LoRA (retains a new pair for each task), and a sequential OSRM adaptation that combines merged LoRA with an initialization derived only from completed tasks.
Shared LoRA Implementation
Implementations of low‑rank LoRA variants and their training pipeline.
This section details how low‑rank LoRA adapters are instantiated and combined across tasks, covering four concrete variants and the overall training pipeline.
A single LoRA adapter is reused for every task, accumulating its updates while keeping the overall rank bounded.
Task 1: optimizer $Opt_1$ learns $A^{\star}_{1}=\begin{bmatrix}1&0\\0&1\end{bmatrix}$, $B^{\star}_{1}=\begin{bmatrix}0.5&0\\0&0.5\end{bmatrix}$.
Effective weight after task 1: $W_{\text{eff}}^{1}=W_{0}+B^{\star}_{1}A^{\star}_{1}=W_{0}+\begin{bmatrix}0.5&0\\0&0.5\end{bmatrix}$.
Task 2 starts with $A^{(0)}_{2}=A^{\star}_{1}$, $B^{(0)}_{2}=0$; $Opt_2$ updates $B^{\star}_{2}=\begin{bmatrix}0&0.3\\0.3&0\end{bmatrix}$ while keeping $A_{2}$ fixed.
Effective weight after task 2: $W_{\text{eff}}^{2}=W_{0}+B^{\star}_{1}A^{\star}_{1}+B^{\star}_{2}A^{\star}_{2}$, still a rank‑$2$ matrix.
Even after multiple tasks the adapter contributes only a rank‑$2$ correction, guaranteeing a compact representation.
How does Shared LoRA differ from the standard LoRA that creates a fresh adapter for each task?
Standard LoRA would allocate a new pair $(A,B)$ per task, causing the total rank to grow linearly with the number of tasks. Shared LoRA re‑uses the same adapter, retaining only the latest factors, so the overall rank stays bounded by $r$.
At each task boundary we fold the current LoRA factors into the dense weight matrix and start a fresh adapter, preserving a compact dense model plus one low‑rank adapter.
Merge after task 1: $W^{1}=W_{0}+B^{\star}_{1}A^{\star}_{1}=W_{0}+\begin{bmatrix}0.5&0\\0&0.5\end{bmatrix}$.
Initialize task 2 adapter: $A^{(0)}_{2}$ from Kaiming, $B^{(0)}_{2}=0$.
During task 2, optimizer learns $B^{\star}_{2}=\begin{bmatrix}0&0.3\\0.3&0\end{bmatrix}$ (with $A_{2}$ fixed).
Final merge: $W^{2}=W^{1}+B^{\star}_{2}A^{\star}_{2}$, still a dense matrix plus a rank‑$2$ correction.
Because each new LoRA starts from zero, the dense matrix always reflects the cumulative knowledge, and the adapter never interferes with past updates.
Why merge the LoRA into the dense matrix instead of simply continuing to train the same adapter?
Merging discards the old adapter’s parameters, preventing them from accumulating stale gradients. It also keeps the total number of stored low‑rank parameters constant (one fresh adapter per task), which is essential for scalability.
Each task receives its own rank‑$r$ LoRA while all earlier adapters are frozen, and a regularization term pushes the new adapter to differ from the accumulated ones.
Task 3 starts with $A^{(0)}_{3}$ drawn from Kaiming and $B^{(0)}_{3}=0$.
During training, optimizer updates $A^{\star}_{3},B^{\star}_{3}$ while $A_{<3},B_{<3}$ stay fixed.
Perpendicularity penalty $R_3^{\perp}$ is computed as the sum of entrywise $\ell_{1}$ norms of $A_{<3}^{\top}A_{3}$, encouraging $A_{3}$ to be orthogonal to $A_{1}$ and $A_{2}$.
Effective weight after task 3: $W_{\text{eff}}^{3}=W_{0}+B_{<3}A_{<3}+B^{\star}_{3}A^{\star}_{3}$.
By freezing earlier adapters and penalizing alignment, O‑LoRA isolates each task’s representation, mitigating interference.
How does O‑LoRA avoid the interference that plagues naïve sequential fine‑tuning?
O‑LoRA freezes all previously learned LoRA factors, so gradients from the current task cannot modify them. The perpendicularity regularizer further forces the new adapter to occupy a subspace not already used, ensuring each task adds novel capacity.
Before training a new task we compute low‑energy input directions from previously seen tasks and initialise the new LoRA’s projection onto those directions, guaranteeing orthogonality to past features.
Form $H_{<3,j}= \begin{bmatrix}1&0&0\\0&1&0\end{bmatrix}$.
SVD of $H_{<3,j}$ yields singular values $(1,1,0)$ and right singular vectors $V_{\min}=\begin{bmatrix}0\\0\\1\end{bmatrix}$ (the third column).
Compute $\kappa_{3,j}= \frac{1}{2}\bigl((A^{\star}_{3}[1,:])^{2}+(A^{\star}_{3}[2,:])^{2}\bigr)$ (using a standard Kaiming init, $\kappa_{3,j}=1$).
Initialise $A^{(0)}_{3,j}=V_{\min}^{\top}=[0\;0\;1]$, $B^{(0)}_{3,j}=0$.
Because $A^{(0)}_{3,j}\,\bar{h}_{1,j}=A^{(0)}_{3,j}\,\bar{h}_{2,j}=0$, the new adapter starts orthogonal to both previous feature means.
Initializing on the smallest singular vectors guarantees that the new LoRA cannot immediately re‑use directions already captured by earlier tasks, reducing redundancy.
Why initialise the new LoRA on the smallest singular vectors rather than the largest?
The smallest singular vectors span directions with the least energy in the stored feature means, i.e., the null space of $H_{<t,j}$. Projecting onto this subspace ensures the new adapter starts orthogonal to all past means, preventing it from duplicating already‑learned representations.
Load and format the task data $D_t$.
If $t>1$, construct teachers for previous states and generate the fixed‑budget replay set when the data anchor is active.
Snapshot the trainable coordinates if the SI anchor is active.
Optimize Equation 14 on task $t$ using the active low‑rank allocation rule.
Consolidate SI and estimate the online‑EWC Fisher if those anchors are active.
Collect the feature means $\{\bar{h}_{t,j}\}$ for sequential OSRM if it is active.
If Shared LoRA is selected, retain $(A^{\star}_{t,j},B^{\star}_{t,j})$ for the next task.
Else if Merged LoRA is selected, commit the merge (Equation 39), attach a fresh rank‑$r$ workspace, save the current factors, and reset the workspace.
Else if O‑LoRA is selected, commit the merge (Equation 39), attach a fresh rank‑$r$ workspace, and apply the sequential OSRM step (Equation 47).
Otherwise, commit the merge (Equation 39), attach a fresh workspace, and apply the sequential OSRM step (Equation 47).
**Table 3.** Canonical implementation and optimization settings.
Search Procedure Details
Ablation analysis of the task‑level successive halving search and its cost savings.