Safin-1: Safety from Within through Memory-Native State Evolution

Ming Zhang, Kaisen Yang, Shu Yu, Ermo Hua, Zhekai Chen, Cheng Jin, Jingnan Zheng, Yi Zhang, Zhongtian Ma, Jiawei Zhou, Sirui Chen, Qiaosheng Zhang, Xiang Wang, Ning Ding, Xia Hu, Bowen Zhou, Youbang Sun, Chaochao Lu

Safin-1 uses routable recurrent state anchors to enable long-horizon memory and persistent safety specialization.

How can we build a recurrent model that uses an addressable, routable memory bank to achieve both long-horizon information retention and intrinsic safety?

Foundation models struggle to maintain long-horizon continuity because they either rely on expensive token-level caches or compress history into a single, easily overwritten recurrent state. Safin-1 introduces Memory-Anchor Routing across Context History (MARCH), which periodically checkpoints the model's recurrent state into an addressable bank, allowing tokens to selectively retrieve relevant historical snapshots. This architecture enables a persistent "Safety State" that can be learned and attached to the model, providing safety specialization without modifying the frozen backbone or relying on external safeguards.

Paper Primer

The core mechanism hinges on treating the model's own recurrent state trajectory as a searchable database. By inserting learned anchor embeddings at fixed intervals, the model creates a bank of state snapshots that the router can query based on current context, effectively decoupling long-term memory from the immediate recurrent update.

Persistent Safety States provide superior safety-utility trade-offs compared to standard adapter-based fine-tuning.

In 35B-A3B scale experiments, the Safety State reduced jailbreak attack success rates by 52.3% while maintaining lower over-refusal rates than a training-matched LoRA control.

MARCH architecture consistently improves general reasoning and long-context retrieval across multiple recurrent backbones.

In 0.8B controlled studies, MARCH outperformed the strongest recurrent baseline on all eight commonsense benchmarks and improved LongBench performance by 25%.

Why use state-level routing instead of standard sparse attention?

Standard sparse attention operates on token-level keys, which requires maintaining a massive cache. Safin-1 retrieves compact, matrix-valued recurrent state snapshots, which are more memory-efficient and native to the model's recurrent computation.

What is the scope of the "Safety from Within" approach?

It is currently an architectural exploration where safety is instantiated as a persistent, learnable state in the memory bank. It is designed to be detachable, allowing for safety specialization without rewriting the shared general-purpose backbone.

Safin-1 demonstrates that recurrent models can achieve long-horizon memory and persistent behavioral specialization by treating their internal state trajectory as an active, addressable substrate rather than a passive compression of context.

Introduction and Motivation

We expose the need for addressable, persistent memory to embed safety directly within model computation.

Long‑horizon tasks demand models that both retain extensive context and reliably invoke appropriate capabilities. Existing designs treat context and safety as separate, transient mechanisms, limiting persistent specialization.

To make safety a first‑class, intrinsic property, models must store and retrieve capability states the same way they preserve contextual information.

**Figure 1.** Safin-1 architecture and representative results. Left: results from 0.8B architectural validation and 35B-A3B scaling. The 4B safety panel compares Qwen3.5 with Safin-1 equipped with its learned Safety State. Right: the Safin-1 architecture routes context-derived and persistent memory states for selective readout.

The key shift is turning transient recurrent states into addressable memory, enabling safety and other capabilities to be stored and invoked natively.

The MARCH State-Routing Backbone

MARCH routes recurrent states through addressable anchors, enabling persistent capabilities and efficient retrieval.

Recurrent models treat hidden memory as a fleeting summary, which makes it hard to reuse specific past states for specialized functions.

MARCH turns the transient recurrent hidden vector into a bank of addressable state anchors, letting each token route to a stored snapshot or a learned capability state.

How does MARCH differ from standard attention mechanisms?

Standard attention treats every token as a query over all other tokens, producing a dense $O(n^2)$ matrix. MARCH instead stores a compact set of matrix‑valued anchors and routes each token to one of them (or to a null option), turning the retrieval into a $O(n\cdot M)$ operation where $M\ll n$ and the heavy matrix readout is performed only for the selected anchors.

State anchors are checkpointed snapshots of the recurrent hidden matrix that live at learned boundaries, providing a stable reference for later routing.

Insert the shared anchor $\xi_1$ after token 2, creating anchor $A^{(1,1)} = S^{(1)}_{2}$.

Apply $f_{\text{out}}$ (identity for simplicity) and residual to obtain $u^{(1)}_1 = A^{(1,1)}$.

Project with $W^{(1)}_k = I$ to get $k^{(1)}_1 = u^{(1)}_1$; similarly $q^{(1)}_1 = u^{(1)}_1$.

For token 4, the candidate set $V_4 = \{1\}$, so the routing query $\rho_4$ scores only $k^{(1)}_1$.

The routing probability $\pi_{4,1}=1$ (null gets zero mass), and the retrieved state contributes $A^{(1,1)} q_4$ to the output.

Even with a single anchor, the model can retrieve the exact matrix snapshot from two steps earlier, demonstrating how MARCH preserves and reuses past hidden information.

Why can the same anchor produce different routing keys across layers?

Because the anchor representation $u^{(\ell)}_m$ is fed through the recurrent update before each layer, its content reflects the cumulative hidden state up to that point. The linear projection $W^{(\ell)}_k$ therefore maps different hidden matrices to different keys, even though the anchor token $\xi$ is identical.

Each token queries the set of previously stored anchors, optionally falling back to a null option, and blends the retrieved matrix‑valued states with its own recurrent update.

Why does the null option share the same softmax instead of a separate gating mechanism?

The null logit is part of the same softmax, so its probability directly competes with real anchors. This means the model can smoothly attenuate retrieval by allocating more mass to null, without needing an extra binary gate that would break differentiability.

Beyond dynamic anchors, the model learns a small set of fixed matrices (e.g., a Safety State) that are always reachable, providing specialized functionality from the start of a sequence.

Compute routing scores for the dynamic anchor (dot product) and for $P$ using the same query $\rho_3$.

Include the null logit $n_3$.

Softmax yields probabilities, e.g. $\pi_{3,\text{anchor}}=0.4$, $\pi_{3,P}=0.5$, $\pi_{3,\varnothing}=0.1$.

Weighted sum adds \$0.4\,A^{(1,1)}q_3 + 0.5\,P q_3$ to the recurrent readout.

Persistent states can dominate the retrieval even when few dynamic anchors exist, because they are always present and can be learned to encode task‑specific knowledge.

What would happen if we omitted the persistent capability states?

Without them the model could only retrieve information that already appeared in the input, limiting its ability to inject specialized functionality (e.g., safety constraints) at the very first token.

High‑level producer–reader loop for a single recurrent layer.

By separating production (checkpointing) from reading (routing), MARCH achieves both dense recurrent modeling and efficient sparse retrieval, enabling large‑scale sequences to be processed with modest memory.

Architectural Validation of MARCH

MARCH consistently improves recurrent model performance across benchmarks.

We isolate MARCH’s architectural contribution by running controlled 0.8 B‑parameter pre‑training studies across several recurrent backbones while keeping data, token budget, sequence length (16 K), and optimizer identical. Four capability groups are then evaluated: zero‑shot commonsense, LongBench long‑context understanding, NIAH associative recall, and real‑world in‑context retrieval.

MARCH achieves the highest average score of 41.5 on eight zero‑shot benchmarks, surpassing the best Gated DeltaNet variant by 1.4 points.

Table 1 reports MARCH avg = 41.5 versus 40.1 for GDN and 40.0 for its Log‑Linear variant; it also slightly exceeds the full‑attention baselines.

MARCH raises the overall LongBench average to 14.9, a 25 % improvement over Gated DeltaNet’s 11.9, and is best on every task.

Table 2 shows MARCH avg = 14.9 versus 11.9 for GDN and 12.5 for Log‑Linear; the row is bolded for all categories.

MARCH improves the average in‑context retrieval score to 23.3, yielding 8 %–23 % gains over the strongest recurrent baseline.

Table 3 lists MARCH avg = 23.3 versus 20.5 for the best GDN variant, with per‑dataset improvements highlighted in green.

MARCH outperforms the strongest recurrent baseline in 19 of 24 NIAH settings, including all multi‑needle tasks at 32 K context length.

Figure 3 and the accompanying analysis report MARCH leading in 19/24 configurations, with pronounced gains on multi‑needle tasks beyond the 16 K training limit.

**Figure 3.** NIAH performance on three single-needle and three multi-needle tasks. The Transformer achieves perfect accuracy on both S-NIAH-1 and S-NIAH-2 at context lengths of 4K, 8K, and 16K.

Generality and Efficiency Analysis

Controlled ablations show how MARCH’s routing, anchor interval, and positional choices affect performance and speed.

We now present a suite of ablations that isolate the contributions of MARCH’s routing, anchor interval, and positional encoding choices.

GDN is a recurrent cell that updates its hidden state by gating a delta term—only the change is added, keeping the core state stable across steps.

KDA refines GDN by applying a separate decay factor per channel, allowing each dimension of the hidden state to forget at its own rate.

MARCH improves the needle‑in‑a‑haystack (NIAH) metric in 11 of the 12 backbone‑length settings.

Table 4 shows consistent gains for GDN, KDA, and GDN2 when MARCH is added.

MARCH also raises SWDE and FDA scores while largely preserving short‑context perplexity and training loss.

Same Table 4 entries for the “In‑Context Retrieval” columns and the “Train Short‑Context” column.

Anchor interval C = 512 yields the best overall trade‑off between retrieval quality and routing cost.

Table 6 (panel a) reports the highest NIAH average for C = 512; panel b shows that the same checkpoint can be run with a denser interval at inference.

**Figure 4.** Training efficiency across sequence lengths. Left: end-to-end training throughput in tokens per second (higher is better). Right: forward-backward runtime of the core sequence-mixing operation in milliseconds (lower is better). MARCH (Top-4) retains only the four highest-scoring state anchors for each token and head during historical retrieval.

MARCH’s sparse routing, a well‑chosen anchor interval, and the optional null slot together deliver consistent accuracy gains while scaling efficiently to very long contexts.

Safin-1 at Scale

Safin‑1 routes internal states via MARCH, turning transient memory into addressable, persistent specialization.

Safin‑1 builds on the MARCH routing bank to make recurrent memory addressable, enabling persistent capability and safety specialization.

Safin‑1 raises GPQA‑Diamond by 6.56 points at the 35B‑A3B scale.

Table 8 shows GPQA‑Diamond improving from 64.65 % to 71.21 %.

A persistent slot in the MARCH bank that stores safety‑aligned behavior, allowing the model to enforce safe responses while the main backbone stays frozen.

How does the Safety State differ from a standard LoRA adapter?

The Safety State lives inside the addressable MARCH bank and persists across contexts, whereas LoRA adds low‑rank updates to weight matrices and does not leverage the routed‑state mechanism.

**Table 8.** Capability and safety results after the matched CPT-SFT pipeline at the 4B and 35B-A3B scales. All values are percentages. Bold denotes the better result within each scale; ties are bolded for both models. ASR and ORR are lower-is-better, whereas all other metrics are higher-is-better.

**Table.** Comparison of Safin-1 (4B) and Safin-1 (35B-A3B) across Safety Robustness, Over-refusal, and Capability Retention benchmarks. Note. Base is the unadapted Safin-1 checkpoint. Averages are unweighted means across the five safety or four capability benchmarks, respectively. Bold compares Safety State with LoRA within each scale; ties are bolded for both. All values are percentages.

Related Work

Key prior approaches to efficient long‑context modeling and safety alignment.

Long‑context sequence modeling has spurred a spectrum of efficient attention and recurrent‑state techniques. This section surveys the most relevant families, highlighting how they differ from the addressable state routing of Safin‑1.

The Transformer processes an entire token sequence in parallel, letting each token attend to every other via learned query‑key‑value interactions.

Approximates the softmax kernel with random feature maps, turning the quadratic attention computation into a linear one.

Combines local sliding windows with a small set of global tokens that attend everywhere, achieving sub‑quadratic cost.

Factorizes the attention graph into a sparse pattern, limiting each query to a subset of keys.

Clusters tokens by content and routes each query to a small set of representative keys.

Assigns queries to relevant key–value blocks via a trainable routing network, enabling block‑wise sparse attention.

Compress the causal prefix into a fixed‑size linear state, updating it recurrently and reading from it via convolution‑like operations.

Combines recurrent updates with associative memory, allowing each token to retrieve from a growing set of past states.

Demonstrates that a few hundred curated safety demonstrations can substantially improve model refusal behavior.

Intervenes directly on hidden‑state activations to steer model behavior without altering backbone weights.

Reroutes internal representations to prevent unsafe activations, acting as a safety‑focused gating layer.

Learns a short input prefix that conditions the model toward safe behavior, effectively a learned prompt.

Across these families, Safin‑1’s MARCH mechanism uniquely preserves the native recurrent‑state trajectory while making each historical snapshot addressable, contrasting with methods that replace or augment the state with external memories.

Discussion and Conclusion

We discuss Safin‑1’s memory routing benefits, safety specialization, limitations, and future directions.

Safin‑1 shows that recurrent state can act as an addressable long‑horizon memory substrate via MARCH routing. Consistent gains on GDN, KDA, and GDN2, and on 4B and 35B‑A3B experiments, confirm that historical‑state access complements the recurrent update rule. Thus, memory quality depends on both writing new information and preserving accessible earlier states, with state anchors serving as compressed cumulative memories rather than token‑level records.

The routed state bank can store both context‑derived anchors and learned persistent capability states, enabling a Safety‑from‑Within specialization. With the backbone frozen, this safety state improves the safety‑over‑refusal trade‑off compared to a training‑matched LoRA, yet its detachability demands protection of both the persistent state and its routing path. Whether multiple persistent states can coexist without routing conflicts remains an open question.

Architecturally, the state bank grows with context length and sparse routing only reduces read cost, not storage, while fixed‑interval anchoring may keep redundant checkpoints. Empirically, our evidence is limited to two Qwen3.5 configurations under a single CPT‑SFT pipeline and a small English safety corpus. Future work should explore adaptive anchor creation, broader model families, and composable persistent states without routing interference.

Safin‑1 treats model state as both addressable memory and a modular interface for capabilities, with 0.8B, 4B, and 35B‑A3B experiments confirming persistent gains. Safety State reduces average ASR on five jailbreak benchmarks by 42.3% and 52.3% at 4B and 35B‑A3B, and yields lower XSTest ORR than rank‑8 LoRA controls. These findings provide early architectural evidence for Safety from Within, motivating further exploration of alternative safety‑representing architectures.

Project Contributors

List of authors and their contribution roles.

The Shanghai Artificial Intelligence Laboratory led the work; authors are ordered by contribution, with * marking equal contributors, ‡ the technical lead, and † the project lead.

The image displays a list of names arranged in a grid format, representing the authors of a research paper.

Training Configurations and Details

Detailed configurations for the small‑scale study, large‑scale models, and safety specialization.

This appendix enumerates the exact architectural, data, and optimization settings used in the controlled small‑scale study, the full‑scale Safin‑1 runs, and the Safety State specialization experiment.

Section A.1 reports the training regime for the small‑scale validation of MARCH, including token budget, sequence length, optimizer hyper‑parameters, and the recurrent backbone details.

Section A.2 details the large‑scale training pipelines for the 4B and 35B‑A3B Safin‑1 models, covering initialization, router head counts, routing dimension, and the learning‑rate schedule.

Section A.3 compares the persistent Safety State against a rank‑8 LoRA control, using identical adaptation protocols and training budgets.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers