Continually Post-Training Small Models via Synthetic Self-Distillation

Woosung Koh, Hyunsoo Lee, Kyungjae Lee, Haeju Park, Dahyun Lee, Dasol Hwang, Moontae Lee

Synthetic Self-Distillation (SYNTHSELF) enables effective continual learning in small models by outsourcing high-quality, in-distribution supervision to larger, same-family aligners.

How can we continually fine-tune small language models on new tasks without losing their original capabilities?

Small language models struggle with continual post-training because they lack the instruction-following capacity to generate the high-quality, in-distribution demonstrations required to prevent catastrophic forgetting. The authors introduce Synthetic Self-Distillation (SYNTHSELF), which replaces on-policy self-supervision with off-policy distillation using a minimally larger, same-family aligner. This aligner generates high-quality, in-distribution rewrites of training data, which are then filtered via rejection sampling to provide clean supervision for the smaller student model. Across three base model scales and four acquisition domains, SYNTHSELF consistently outperforms standard fine-tuning and existing distillation methods in both new capability acquisition and retention of prior knowledge.

Paper Primer

The core mechanism hinges on selecting a "minimally larger" aligner from the same model family to act as a teacher. This aligner is like a senior mentor who speaks the same technical dialect as the student: it provides high-quality, verified reasoning traces that remain within the student's native distribution, avoiding the performance degradation caused by out-of-distribution targets.

SYNTHSELF operates as an off-policy pipeline: generate synthetic rewrites, filter them for correctness via rejection sampling, and train the student on the resulting high-quality dataset. This approach decouples supervision quality from the student's own limited reasoning abilities.

SYNTHSELF significantly improves acquisition of new capabilities in small models compared to on- and off-policy distillation.

Averaged across four settings and three base model scales (0.5B, 1.5B, 3B), SYNTHSELF outperforms distillation baselines by +5.7 to +16.9 percentage points.

Using a same-family aligner is more efficient than using larger, out-of-distribution models.

Selecting a same-family aligner yields a 69.9% reduction in alignment computational cost (FLOPs) while maintaining superior acquisition performance.

Why does this method use an external aligner instead of the model's own self-distillation?

Small models lack the in-context learning and instruction-following abilities to generate high-quality self-supervision. The authors show that small-model self-generated traces are often inaccurate or incomplete, leading to poor acquisition.

What is the primary assumption behind the success of this approach?

The authors assume that staying "in-distribution" is the critical factor for continual learning. They demonstrate that forgetting correlates linearly with the distribution shift of the policy, making it essential to use targets that align with the base model's native distribution.

For practitioners deploying small models, SYNTHSELF provides a robust, tuning-free recipe for continual learning that avoids the pitfalls of catastrophic forgetting by prioritizing distribution-aligned, high-quality synthetic supervision.

Motivation and Problem Framing

We expose why self‑distillation fails for small language models and set up the synthetic rewrite solution.

Continual post‑training must both acquire new capabilities and retain existing ones, yet small language models repeatedly lose prior knowledge—a phenomenon known as catastrophic forgetting. Self‑Distillation Fine‑Tuning (SDFT) leverages a model’s own in‑context learning and instruction‑following to generate self‑supervision, but this on‑policy approach collapses on smaller models because they lack the fluency required to produce reliable demonstrations, making the rewrite process much harder than on larger models.

When a model is fine‑tuned on new data, it can overwrite the parameters that encoded its previous knowledge, causing the model to forget what it once knew.

SDFT lets a model teach itself by generating in‑context demonstrations from its own knowledge and then distilling those demonstrations back into itself, effectively turning the model into both teacher and student.

Empirically we find that the amount of distribution shift experienced by the post‑trained base model predicts both poor acquisition and the degree of forgetting, establishing in‑distribution learning as the key to successful continual training.

**Figure 3.** **SDFT fails at smaller model sizes as its ability to self-teach deteriorates.** (a) SDFT acquisition performance relative to SFT declines monotonically as model size decreases. (b) For example, in the tool-use domain, self-rewriting results in reduced accuracy and missing CoTs.

The failure of existing self‑distillation at small scales stems from inadequate in‑distribution supervision, motivating the need for a synthetic rewrite approach.

The SYNTHSELF Mechanism

Shows how SYNTHSELF keeps new data in‑distribution while delivering high‑quality supervision.

Small language models lose prior capabilities when fine‑tuned with standard self‑distillation (SDFT), because the generated targets drift out of the base model’s distribution.

SYNTHSELF replaces on‑policy self‑distillation with an off‑policy rewrite step that stays in‑distribution and then filters rewrites for correctness.

How does SYNTHSELF differ from the standard SDFT pipeline?

Standard SDFT trains on‑policy outputs, which can drift the student away from the base distribution. SYNTHSELF generates rewrites off‑policy with a stronger aligner, then discards any rewrite that changes the answer, so the training data remain in‑distribution while still being higher quality.

Generate $\hat{y}_1$ by greedy decoding from $\pi_A$ under $P_{\text{rewrite}}(x_1, y_1)$ → “The half‑life is 5.73 × 10³ yr”.

Extract $a(\hat{y}_1)=\text{A}$, which matches $a(y_1)$, so add $(x_1,\hat{y}_1)$ to $\hat{D}$.

Generate $\hat{y}_2$ → “The half‑life is 4.56 × 10³ yr”.

Extract $a(\hat{y}_2)=\text{B}$, which matches $a(y_2)$, so add $(x_2,\hat{y}_2)$ to $\hat{D}$.

Run SFT on $\hat{D}$ for one epoch, updating the student parameters.

Rejection sampling guarantees that every synthetic example preserves the original answer, eliminating label noise while still providing richer phrasing.

An In‑Distribution Aligner is a larger model selected to produce rewrites that stay close to the base model’s token distribution.

Why not simply use the base model $\pi$ itself as the aligner?

The base model is already the student; using it to generate rewrites would reproduce the same distribution drift that SDFT suffers from. A larger, higher‑quality aligner can produce richer, more accurate rewrites while still being close enough to the base model to keep the data in‑distribution.

Compute KL between $\pi$ and each $\pi_A^{(k)}$’s rewrite distribution $q(y)$. Results: 0.12, 0.08, 0.07.

Compute $R(\pi_A^{(k)})$ (answer‑match rate). Results: 0.94, 0.96, 0.97.

Select $\pi_A^{(3)}$ because it has the smallest KL (0.07) and the highest $R$ (0.97).

The best aligner is not the smallest model; a larger model can stay closer to the base distribution while delivering more correct rewrites.

Collect the original demonstration set $D_{\text{train}}=\{(x_i,y_i)\}$.

Pick the In‑Distribution Aligner $\pi_A^*$ according to Eq. (6).

For each $(x_i,y_i)$, generate a deterministic rewrite $\hat{y}_i = \arg\max_y \pi_A^*(y\mid P_{\text{rewrite}}(x_i,y_i))$.

Apply rejection sampling: keep $(x_i,\hat{y}_i)$ only if $a(\hat{y}_i)=a(y_i)$.

Aggregate all kept pairs into the offline dataset $\hat{D}$.

Perform standard supervised fine‑tuning on $\hat{D}$ to obtain the updated student $\pi_{\hat{\theta}}$.

**Figure 4.** Same-family rewrites provide more in-distribution off-policy data relative to Gold SFT. Conversely, stronger, cross-family models fail to meaningfully improve distributional alignment with the base model.

**Algorithm 1: SynthSelf (post-aligner selection)** **Input:** Demonstrations $D_{train} = \{(x_i, y_i)\}_{i=1}^N$, post-trained base model $\pi_\theta$, aligner $\pi_A^*$, rewrite instruction $P^{rewrite}$, answer extractor $a(\cdot)$ 1. $\hat{D} \leftarrow \emptyset$; // Initialize offline dataset 2. **foreach** $(x_i, y_i) \in D_{train}$ **do** /* Synthetic In-distribution Rewrite */ 3. $\hat{y}_i \leftarrow \arg \max_y \pi_A(y | P^{rewrite}(x_i, y_i))$; // Greedy decoding /* Rejection Sampling */ 4. **if** $a(\hat{y}_i) = a(y_i)$ **then** 5. $\hat{D} \leftarrow \hat{D} \cup \{(x_i, \hat{y}_i)\}$; // Same answer, but in-distribution 6. **else** 7. discard $(x_i, \hat{y}_i)$ 8. **end** 9. **end** /* Off-policy learning via SFT */ 10. $\hat{\theta} \leftarrow \arg \min_\theta \mathbb{E}_{(x, \hat{y}) \sim \hat{D}} [-\log \pi_\theta(\hat{y} | x)]$; 11. **return** $\pi_{\hat{\theta}}$;

Experimental Results and Analysis

Empirical results showing SYNTHSELF’s acquisition gains and near‑zero forgetting across domains.

We evaluate SYNTHSELF across three domains and a science mixture to assess acquisition and retention, then probe distribution matching and aligner effects.

When the data used to fine‑tune a model diverges from the model’s original training distribution, the model’s performance on previously learned tasks degrades.

SYNTHSELF improves acquisition by up to +16.9%p while preserving retention comparable to the strongest baselines.

Table 1 shows SYNTHSELF leading acquisition across all four benchmarks and matching the best retention scores on 14 tasks.

SYNTHSELF outperforms SDFT by +18.96%p on acquisition across domains.

Figure 6 reports a +18.96%p acquisition gain for SYNTHSELF over prompt‑matched SDFT.

**Figure 1.** (a) Averaged across 3 base models, continual post-training via SYNTHSELF on the medical domain results in near-zero forgetting across all 14 benchmarks. (b) Across 3 base models, 4 acquisition benchmarks, and 14 retention tasks, SYNTHSELF maximizes performance gains with minimal forgetting.

**Figure 2.** Distribution shift negatively correlates with ideal continual learning. (a) Experiment 1: Post-training distribution shift strongly correlates with forgetting existing capabilities across GPQA-Diamond, ARC-C, MMLU-Pro, HumanEval, and MBPP, regardless of data source or type. (b) - (e) Experiment 2: Transforming the same dataset from its original OOD form (HI) to more in-distribution forms (MID, LO) monotonically improves (c) target domain (math) retention, (d) target domain performance, and (e) cross-domain (coding) retention.

**Figure 6.** **SYNTHSELF** outperforms **SDFT** across diverse prompt variations, including a version prompt-matched to **SYNTHSELF**. On average, **SYNTHSELF** outperforms prompt-matched **SDFT** by +18.96%p and +1.68%p on acquisition and retention, respectively.

SYNTHSELF achieves near‑zero forgetting across diverse domains.

Implementation Details and Limitations

Limits, compute savings, and future directions for SYNTHSELF.

Related work on distribution drift in SFT (Huang et al., 2026) and combined mechanisms (Li et al., 2026) is noted, but neither addresses continual learning for small models.

Practitioners can increase the pass@k parameter, trading additional inference compute for broader coverage supervision.

SYNTHSELF’s main limitation is the need for a larger same‑family aligner; most open‑weight families already include a hierarchy of model sizes.

Future directions include adapting out‑of‑family aligners and prompting the base model’s answer in‑context to generate in‑distribution rewrites.

**Table 3.** The training data. The 23 corpora Experiment 1 trains on, each taken as published and named by the HuggingFace path it is loaded from. Groupings are domain and upload date against the base model's 2024-09 cutoff, and the dates are each repository's createdAt read from the HuggingFace API.

**Table 4.** Train/test split of the SCIENCE-MIX corpus. SCIKNOWEVAL L-3 gold, split 80/20 at seed 0.

**Table 5.** Configuration for every cell in Tabs. 1 and 2. Unlisted values are library defaults.

AI was used for code generation, grammar review, and generating reasoning demos (Chemistry demos with GPT‑4o); it was not used for conceptual design, data analysis, or result interpretation.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers