Spectral Rewiring for Exploration, Purification, and Model Merging

Zhilong Zhang, Hongli Yu, Huan-ang Gao, Hanlin Wu, Yuxuan Song, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou

Subspace-Aligned Rewiring (SAR) extracts reasoning-effective updates by projecting RL-trained weights onto a base model's spectral manifold.

How can we compress dense reinforcement learning updates into a compact, spectral-aligned subspace to preserve reasoning gains while avoiding the performance degradation typical of full-parameter updates?

Full-parameter reinforcement learning (RL) often suppresses reasoning performance and creates interference when consolidating multiple capabilities, as dense updates lack geometric control. The authors propose Subspace-Aligned Rewiring (SAR), which projects RL updates onto the base model's pretrained singular-vector basis. This method isolates a compact "rewiring matrix" that captures how pretrained skills are logically connected, while discarding residual noise. SAR preserves over 99% of post-training performance using as little as 0.6% of total parameters, while simultaneously improving exploration and cross-domain merging.

Paper Primer

SAR treats the base model's singular value decomposition (SVD) as a fixed functional library. The core move is to decompose the RL update into a rewiring matrix $M$ that maps between these pretrained singular vectors: $M$ acts as a routing layer, enabling the model to synthesize complex outputs from independent latent premises.

SAR extracts a compact reasoning core that matches full-RL performance.

Across 1.5B to 32B parameter models, SAR preserves >99% of peak Pass@1 performance on AIME benchmarks using only ~0.58% to 0.64% of total parameters.

SAR improves cross-domain model merging and exploration.

Merged models surpass the best single-domain experts on math and code benchmarks, while SAR-projected models outperform full-RL baselines in high-k exploration (Pass@k). +2.52% average improvement on agentic coding benchmarks

Why does this approach work better than simple low-rank compression?

Low-rank extraction alone captures parameter efficiency, but SAR's spectral alignment is critical. Ablations show that random projections or diagonal-only rewiring fail to recover the full reasoning gains, confirming that the specific pretrained SVD basis is essential for relational synthesis.

What is the scope of this method?

SAR is most effective when the base model already contains the relevant latent capabilities. It is less compatible when RL requires extensive behavioral rewriting, such as learning new execution protocols or formatting conventions not present in the base model.

SAR demonstrates that reasoning elicitation is primarily a process of reorganizing existing latent skills rather than acquiring entirely new ones, allowing for training-free model editing.

Researchers can now treat RL updates as editable spectral objects, enabling efficient model merging and performance purification without the need for full-parameter retraining.

The Reasoning Bottleneck in RL

RL post‑training updates are dense, hurting reasoning and multi‑domain performance.

Full‑parameter reinforcement‑learning updates touch every weight in a large language model. This dense, unconstrained change creates two practical bottlenecks: reasoning performance collapses, often visible as early saturation of test‑time scaling, and interference when multiple capabilities are merged or jointly trained.

Updating all parameters at once perturbs the delicate weight patterns that encode reasoning, and it mixes unrelated domain signals, so the model’s existing skills can be overwritten or conflicted.

To separate the useful reasoning component from the noisy residual, we need a coordinate system that respects the pretrained model’s internal structure. The singular‑value decomposition (SVD) of the base weights provides such a basis: singular vectors encode atomic skills, while singular values encode their original strengths.

Subspace‑Aligned Rewiring (SAR) projects a dense RL update onto this spectral manifold, keeping the reasoning‑effective core and discarding orthogonal directions that harm performance or cause domain conflict.

**Table.** Performance comparison of single-domain experts and cross-domain merging methods at the 1.5B scale.

The trade‑off between reasoning gains and model stability can be managed by extracting a compact spectral core from post‑training updates.

The Geometric Framework of SAR

Geometric framework projects RL updates onto the pretrained spectral manifold to isolate a compact reasoning core.

Full‑parameter RL updates introduce dense, unconstrained changes that corrupt the pretrained model’s reasoning geometry. By projecting these updates back onto the model’s intrinsic spectral subspace we can recover a compact, reasoning‑effective core.

The pretrained weight matrix defines a set of orthogonal skill directions; these singular vectors form a functional coordinate system that the model uses to read‑in features and read‑out outputs.

Right singular vector $v_{1}=[1,0]^{\top}$ extracts the first input component; $v_{2}=[0,1]^{\top}$ extracts the second.

Left singular vector $u_{1}=[1,0]^{\top}$ scales the first component by $3$; $u_{2}=[0,1]^{\top}$ scales the second by $1$.

The output for input $x=[2,5]^{\top}$ is $y = 3\cdot2\cdot u_{1} + 1\cdot5\cdot u_{2} = [6,5]^{\top}$.

This tiny example shows how the singular vectors act as independent skill axes; any update that stays within their span will merely re‑weight existing skills.

How does the Spectral Manifold differ from a generic low‑rank approximation?

Low‑rank approximation merely reduces parameter count, whereas the Spectral Manifold is defined by the pretrained model’s own orthogonal skill directions. Updates constrained to this manifold preserve the original geometry, which is crucial for retaining reasoning abilities.

SAR extracts the portion of a full‑parameter RL update that lives inside the pretrained spectral manifold, discarding off‑manifold components that tend to degrade reasoning.

Subspace‑Aligned Rewiring (SAR) pseudocode.

Compute $U=V=I_{2}$ (since $W_{0}$ is already diagonal).

Take $k=2$, so $\Delta W_{k} = \Delta W$ (no truncation).

Project: $M = U^{\top}\Delta W_{k} V = \Delta W_{k}$.

Reconstruct aligned update $\Delta W^{*}=U M V^{\top} = \Delta W_{k}$ (identical in this simple case).

Final SAR model $W_{\text{SAR}} = W_{0} + \Delta W^{*} = \begin{bmatrix}2.5 & 0.3\\0.2 & 1.1\end{bmatrix}$, which lies entirely in the original spectral subspace.

Even when the raw RL update contains off‑diagonal entries, projecting with $U$ and $V$ guarantees the result respects the pretrained skill directions, preventing arbitrary distortion of the reasoning geometry.

Is SAR just low‑rank fine‑tuning?

No. Low‑rank fine‑tuning reduces parameter count but still updates arbitrary directions. SAR first projects the update onto the *pretrained* singular basis, preserving the original geometry and only allowing cross‑skill mixing via the dense rewiring matrix $M$.

Off‑diagonal entries $M_{ij}$ let several distinct input skill directions $v_{j}$ jointly drive a single output direction $u_{i}$, enabling compositional reasoning.

Why do off‑diagonal entries enable reasoning while diagonal ones do not?

Diagonal entries only change the strength of an existing one‑to‑one mapping (a single premise to a single conclusion). Off‑diagonal entries create new pathways that combine multiple premises into one conclusion, which is the algebraic analogue of relational inference.

**Figure 1.** Overview of Subspace-Aligned Rewiring (SAR). SAR identifies a compact reasoning core by projecting reinforcement learning updates back onto the pretrained model's spectral manifold. This alignment preserves the intrinsic reasoning basis of the base model while removing off-manifold update components introduced by full RL. The resulting spectral update retains Pass@1 performance, improves large-k exploration, and transfers more robustly across domains, including Mix-RL and expert merging.

**Figure 4** Illustration of spectral rewiring. The base model primarily performs point-to-point associations between isolated spectral components. RL-induced rewiring introduces many-to-one routing, allowing multiple premise directions to jointly activate an output direction associated with a compositional reasoning step.

Validating the Reasoning Core

Single‑Domain Extraction shows SAR keeps full‑RL reasoning with a 1 % spectral rank.

SAR matches full‑RL reasoning performance across model families while retaining only a tiny spectral rank.

Table 1 shows that projected models achieve the same AVG@32 and Pass@32 scores as their full‑RL counterparts within evaluation variance.

DeepScaleR is a reinforcement‑learning fine‑tuned language model that serves as a strong reasoning baseline for the SAR experiments.

How does DeepScaleR differ from a vanilla RL‑fine‑tuned model?

DeepScaleR uses a reward‑shaped objective that explicitly penalizes reasoning errors, whereas a vanilla RL fine‑tune would optimize only token‑level likelihood without the reasoning‑specific signal.

The table compares two models, "DeepScaleR full RL" and "SAR, 1% rank", across three metrics: Coverage@256, Solved / Total, and Gain.

**Figure 5.** Pass@k scaling on AIME 24 among full RL, SAR, and the low-rank component without projection.

SAR preserves full‑RL performance with only 1 % rank updates.

Generalization and Model Merging

Projecting updates onto the spectral manifold boosts cross‑domain performance while preserving reasoning.

Recall that full‑parameter RL updates can degrade reasoning; SAR projects updates onto the pretrained spectral manifold to keep only the reasoning‑effective core.

SAR improves cross‑domain coding performance by up to +1.5 % while preserving math and instruction‑following abilities.

LiveCodeBench v5 rises from 67.61 % (Full RL) to 69.09 % (SAR); AIME 25 Pass@16 changes from 88.33 % to 91.71 %; IFEval moves only from 92.42 % to 92.05 %.

When updates tuned for one task push parameters in directions that hurt another task, the tasks interfere and overall performance drops.

How does SAR differ from simply lowering the learning rate to reduce interference?

SAR selectively projects the update onto the pretrained spectral manifold, discarding task‑specific residual directions while keeping the reasoning‑effective core. Lowering the learning rate uniformly shrinks all directions, including the useful reasoning ones, so it cannot preserve performance as effectively.

**Figure 3.** Cross-domain performance evaluation on the OLMo-3-32B scale. By extracting reasoning-effective parameters, SAR reduces cross-domain interference, improves LiveCodeBench v5/v6, maintains instruction following (IFEval) and math reasoning (AIME 24), and enhances exploration capability.

Summary and Implications

We discuss SAR’s effectiveness, scope, and limits, and interpret cross‑domain merging results.

SAR consistently preserves or improves key downstream metrics when applied to single‑domain extraction, Mix‑RL purification, and expert model merging.

This pattern supports the core insight that most reasoning‑effective change from RL is not random drift but a compact rewiring of capabilities already encoded in the pretrained spectral geometry.

Consequently, SAR is expected to shine when the base model already harbors the target reasoning skill or domain knowledge, and RL mainly elicits or reorganizes that latent ability.

The primary limitation is projection compatibility: the extent to which an RL update can be expressed as a re‑arrangement of knowledge already present in the reference model.

In heavily optimized reasoning RL, projecting the JustRL update after more than 4000 steps still yields an AIME 2024 score around 43 %, comparable to DeepScaleR‑level reasoning but short of the final RL model’s performance.

This gap indicates that while the spectral component captures substantial elicitation, the remaining improvement likely comes from task‑specific rewriting that lies outside the reference manifold.

Direct code‑RL from a base model is less projection‑compatible because the update must acquire new code formatting, tool‑interface behavior, and execution conventions rather than merely recombining existing knowledge.

Once a code‑specific supervised fine‑tuning stage establishes the necessary domain knowledge, the subsequent RL update aligns better with spectral projection, improving compatibility.

PPO training with an external critic introduces a distribution shift between actor and critic, which can generate preferences that are poorly aligned with the reference model’s spectral manifold.

Across both 1.5 B and 14 B scales, SAR’s projected merging consistently outperforms naive interpolation, delivering higher math and code scores and better average performance.

Ablations and Related Work

Component ablations reveal why spectral alignment drives SAR’s gains.

We isolate the contribution of each SAR ingredient by swapping it out for a control while keeping the 1 % low‑rank budget constant.

**Table 7.** No-projection control for 1.5B math-code merging. The no-projection merge uses the same 1% low-rank math update budget as SAR but skips alignment to the pretrained spectral basis. It preserves code performance but loses much of the math gain, whereas SAR achieves positive-sum transfer.

SAR solves one additional AIME 2024 problem at Pass@256 that neither Full RL nor the no‑projection control reaches.

Table 7 shows SAR’s Pass@256 solving 1 more problem than the other methods under the same 256‑rollout budget.

Across domains, the same pattern holds: low‑rank extraction alone preserves single‑sample performance but fails to sustain the scaling advantage that SAR provides.

**Table 8.** Method ablations on DeepScaleR-1.5B with a 1% update budget on AIME 2024. SAR preserves full-RL-level Pass@1 while maintaining Pass@32, whereas random projection and partial rewiring controls lose reliability or coverage.

Full RL retains the highest Pass@1 at 40.31 %, while SAR matches its Pass@32 despite using only a 1 % spectral budget.

Table 8 reports Full RL Pass@1 = 40.31 % and SAR Pass@32 equal to the Full RL value.

These ablations confirm two design insights: (1) the pretrained singular‑vector basis is essential—random projections fall short; (2) both diagonal scaling and off‑diagonal rewiring are needed—each alone recovers only a subset of the reasoning signal.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers