SpecMRL: Redesign Matryoshka Representation Learning within Parametric Spectrum
Zhongxin Chen, Songhao Wu, Heng Cui, Cong Li, Xiao Zhou, Rui Yan
SpecMRL replaces fixed-basis prefix truncation with a learnable spectral basis to optimize nested representation quality.
How can we improve Matryoshka Representation Learning (MRL) by replacing the standard coordinate-based prefix selection with a learned spectral basis?
Matryoshka Representation Learning (MRL) allows models to output embeddings of varying sizes by truncating the leading dimensions, but it lacks a principled way to ensure those early dimensions contain the most useful information. SpecMRL solves this by replacing the static coordinate system with a learnable spectral basis derived from a projection matrix, which organizes features by importance. This approach consistently outperforms standard MRL across all compression ratios, achieving superior performance with only half the embedding dimensions.
Paper Primer
SpecMRL treats the representation space as a learnable spectral organization: it projects the backbone output onto an ordered basis derived from the spectrum of a learned Gram matrix. This acts like a smart filter—it reorders the feature space so that the most informative directions are always pushed to the front, ensuring that any truncated prefix is the most compact, high-quality representation possible.
SpecMRL extends the performance frontier of MRL, surpassing full-dimensional MRL performance using only half the dimensions.
Empirical evaluation on MTEB benchmarks across multiple model families (Qwen3, Llama3). Up to 3.80 points improvement over MRL under 16× compression.
The spectral basis design is robust and efficient, incurring negligible training overhead.
Throughput degradation is < 1% compared to vanilla MRL, significantly outperforming standard eigen solvers.
Why is a learnable spectral basis superior to simply training with standard MRL?
Standard MRL relies on a heuristic prior that assumes the leading dimensions are naturally informative, but it provides no explicit constraint to enforce this. SpecMRL explicitly organizes the feature space into an ordered orthogonal basis, ensuring that the most informative directions are prioritized for nested readouts.
Does this method require changing the underlying model architecture?
No. SpecMRL is a drop-in modification that only changes the construction of the nested representation basis; the backbone model and the training pipeline remain otherwise identical to vanilla MRL.
Introduction and Motivation
Identifies the limitation of coordinate‑based prefix selection in MRL and motivates SpecMRL.
Standard Matryoshka Representation Learning (MRL) simply truncates the full embedding to obtain nested prefixes, assuming the earliest coordinates already contain the most useful information. In practice this coordinate‑based heuristic often misaligns with the true importance of features, limiting compression quality.
MRL reads out prefixes directly from the original basis $\{e_i\}$, which does not guarantee that the first few dimensions capture the most informative features for compact representations.
The limitation of coordinate‑based prefix selection in MRL is that it assumes the original axes already prioritize information, which often fails and hampers effective compression.
Matryoshka Representation Learning
Provides a concise overview of MRL’s contrastive training and nested‑dimension formulation.
We first describe how large language models produce sentence embeddings and the contrastive objective that trains them. Then we introduce Matryoshka Representation Learning, which extends this objective over nested dimensions.
MRL treats a representation as a hierarchy of prefixes, keeping the first $m$ entries of the embedding as a compressed view and training each prefix with its own contrastive loss.
The SpecMRL Mechanism
SpecMRL learns a spectral basis to replace fixed prefixes, improving compression at all levels.
Standard MRL forces nested prefixes to follow the original coordinate axes, which limits how much information can be packed into early dimensions. SpecMRL replaces this rigid ordering with a learned spectral basis, allowing the model to concentrate information where it matters most.
SpecMRL swaps the fixed coordinate‑wise prefix readout for a projection onto the leading eigenvectors of a learned Gram matrix, so the most informative directions become the first $m$ dimensions.
Compute eigenvectors of $G$: they are the standard basis vectors $e_1, e_2, e_3$ with eigenvalues $1,4,9$.
Order them descending → $U = [e_3, e_2, e_1]$ (columns $[0,0,1]$, $[0,1,0]$, $[1,0,0]$).
Pick $m=2$ leading columns $U_{:,1:2} = \begin{bmatrix}0 & 0\\0 & 1\\1 & 0\end{bmatrix}$.
Project an embedding $X = (x_1,x_2,x_3)$: $X^{(2)}_{\text{Spec}} = X U_{:,1:2} = (x_3, x_2)$.
Thus the first two spectral dimensions capture the original third and second coordinates, concentrating the larger variance (9 and 4) into the early slots.
The spectral basis reorders dimensions so that the most energetic components appear first, directly addressing the limitation of fixed coordinate ordering.
How does SpecMRL differ from the original MRL coordinate heuristic?
The original heuristic simply assumes earlier dimensions are more important and never adapts; SpecMRL learns an orthogonal rotation that explicitly orders dimensions by captured variance, so the model can place the most informative features in the first slots instead of being stuck with the raw coordinate order.
SpecMRL learns a rotation of the embedding space that aligns the data’s principal directions with the prefix slots, turning a static coordinate system into a data‑driven one.
Initialize $U^{(0)}$ with the eigenvectors of the initial Gram matrix $G^{(0)} = W_{\text{Proj}}^{(0)} {W_{\text{Proj}}^{(0)}}^{\top}$.
For each training step $t$: compute $G^{(t)} = W_{\text{Proj}}^{(t)} {W_{\text{Proj}}^{(t)}}^{\top}$.
Update the basis via $U^{(t)} = \text{QR}( G^{(t)} U^{(t-1)} )$.
Detach $U^{(t)}$ from the gradient graph before the next iteration to keep the update stable.
After training, cache the final $U$ and use it as a fixed rotation at inference time.
SpecMRL training step – warm‑started spectral update, readout, and loss.
Why does the method use QR orthogonalization instead of recomputing eigenvectors each step?
QR is cheap (a single matrix‑multiply plus a stable factorization) and guarantees an orthogonal basis, whereas a full eigen‑decomposition is $O(D^3)$ and numerically unstable when eigenvalues are clustered. The power‑iteration plus QR therefore provides a fast, stable approximation of the leading eigenvectors.
**Figure 2.** Illustration and pseudocode of SpecMRL. (a) Embeddings are rotated into a learned spectral basis before prefix readouts. (b) The basis is updated by one warm-started orthogonal iteration per training step and cached for the next step. `mrl_loss` denotes the original Matryoshka objective.
Empirical Results
SpecMRL consistently beats standard MRL across compression levels.
Recall that standard MRL aligns nested prefixes with the original coordinate axes, while SpecMRL learns a spectral basis to concentrate information into leading dimensions.
SpecMRL improves MTEB scores by +1.07 points over MRL at compression $τ\!=\!2$.
Table 1 shows SpecMRL 61.61 vs. MRL 60.54 on the Qwen3‑0.6B backbone.
**Figure 3.** MTEB performance trends of SpecMRL across nested dimensions. The final spectral band introduces an unexpected performance drop.
**Figure 4.** MTEB subset results evaluated every 100 steps. The advantage of SpecMRL is consistent throughout the training process.
SpecMRL consistently outperforms MRL across all tested dimensions.
Comparison Baselines
Implementation details for ESE and MIPIC, plus ablations and a linear‑readout comparison.
We evaluate two recent MRL‑based techniques—ESE and MIPIC—by attaching them to the final‑layer representations of Qwen3‑0.6B and by measuring how well they preserve information under aggressive dimensional compression.
ESE keeps the original learn‑to‑compress loss but applies it only to the 1024‑dimensional sentence embedding produced by the last transformer layer, discarding any intermediate‑layer supervision.
MIPIC augments the standard MRL objective with a Self‑Importance Alignment (SIA) term that forces token‑level importance distributions to stay consistent across compression levels.
This ablation isolates the effect of the learnable projection $W_{\text{proj}}$ by applying it directly to truncated embeddings, bypassing the spectral basis computation.
**Table 11.** Comparison with learnable readout.
Parameter Space Analysis
Ablation study linking readout subspace alignment to embedding geometry.
We examine how the readout matrix $W_{proj}$ mirrors the geometry of the learned embedding space. This analysis reveals whether the readout basis captures the dominant semantic directions of sentence embeddings.
Measures overlap between the model’s readout basis $U$ and the principal directions $U_{\hat{E}}$ of a large set of calibration embeddings, indicating how well the readout captures the dominant semantic axes.
How does readout subspace alignment differ from the random baseline $k/D$?
The random baseline expects only $k/D$ overlap because a random subspace has no knowledge of the data distribution. In contrast, $S_k$ quantifies the actual alignment achieved by the learned readout basis; values substantially above $k/D$ indicate that training has shaped $W_{proj}$ to capture the dominant semantic directions of the embeddings.
We randomly sample $N = 10^6$ sentences, encode them with both MRL and SpecMRL to obtain calibration embeddings $\hat{E}_{\text{cal}} \in \mathbb{R}^{N \times D}$, and extract their principal directions $U_{\hat{E}}$. We then compute $S_k$ between $U_{\hat{E}}$ and the model’s readout basis $U$ for several values of $k$.
**Table 12.** Readout subspace alignment.
SpecMRL improves subspace alignment over MRL by +5.03 % at $k=64$.
Table 12 reports $S_{64}$ of SpecMRL exceeding MRL by +5.03 %.
SpecMRL improves subspace alignment over MRL by +15.12 % at $k=128$.
Table 12 reports $S_{128}$ of SpecMRL exceeding MRL by +15.12 %.
SpecMRL improves subspace alignment over MRL by +15.44 % at $k=256$.
Table 12 reports $S_{256}$ of SpecMRL exceeding MRL by +15.44 %.
SpecMRL improves subspace alignment over MRL by +3.21 % at $k=512$.
Table 12 reports $S_{512}$ of SpecMRL exceeding MRL by +3.21 %.
**Figure 5.** Average projection energy fractions.
These ablations show that the spectral readout basis of SpecMRL aligns more closely with the dominant embedding subspace, preserving informative structure under dimensional readout and thereby explaining its superior low‑dimensional performance.
Energy Concentration Analysis
Analyzes how dropping final spectral bands concentrates energy and impacts SpecMRL performance.
We partition the learned spectral basis into 20 consecutive bands ordered by eigenvalue and project the calibration embeddings onto each band. Using $N = 10^6$ passages from the MS MARCO corpus, we compute the projection $z_{ij}=u_{j}^{\top}X[i]$ for each embedding $X[i]$ and define the average energy fraction $E_{b}= \frac{1}{N}\sum_{i=1}^{N}\frac{\sum_{j\in B_{b}} z_{ij}^{2}}{\sum_{j=1}^{1024} z_{ij}^{2}}$ per band $B_{b}$. This setup lets us assess how energy is distributed across spectral directions.
The final spectral band captures a disproportionately large share of energy.
Measured average energy fraction $E_{b}$ shows the final band holds 21.72 % of total energy despite covering only 5 % of the spectral directions.
Computational Complexity
Analyzes the extra FLOPs introduced by SpecMRL and shows they are negligible.
This section quantifies the extra computation that SpecMRL adds on top of standard MRL. The overhead comes from building a Gram matrix, updating an orthogonal basis, and projecting a batch of $B_{\text{proj}}$ embeddings of dimension $D$.
Implementation Details
Implementation specifics for training, data, and evaluation of SpecMRL and baselines.
We train SpecMRL and baseline methods on three backbone families—Qwen3‑0.6B‑Base, Qwen2‑0.5B, and Llama‑3.2‑1B—adding Qwen3‑4B‑Base for scaling experiments. Training follows the MRL protocol of Hu et al. (2025), with nested dimensions $m_i$ and loss weights $w_i$ detailed in Table 13. Data come from the public KaLM‑embedding‑finetuning‑data set; we sample 1,152,000 triples (query, positive, seven hard negatives) unless full‑run experiments require the entire corpus.
Training uses a per‑device batch size of 96 (global 384) across four GPUs, sequence length 512, and GradCache to curb memory. We optimize with AdamW (learning rate $10^{-4}$, warm‑up 0.1, linear decay, weight decay $0.01$) and a contrastive temperature of $0.01$. The projection matrix $W_{Proj}$ is initialized from $\mathcal{N}(0,0.022)$; its spectral basis is obtained as the eigenvectors of $W_{Proj}^\top W_{Proj}$.
Evaluation follows the MTEB benchmark, covering 56 datasets in STS, classification, clustering, pair classification, reranking, retrieval, and summarization. For the figures in the paper we report the average over a 15‑dataset subset (Table 14). SpecMRL performs one orthogonal iteration in FP32 on the final Gram matrix, fixing the basis for inference. Embeddings are left unnormalized for classification tasks and L2‑normalized for all other tasks.
**Table 13.** MRL training configurations for different backbone models.
**Table.** Performance comparison of different methods across varying embedding dimensions (1024 to 64) and compression ratios ($\tau$).
Experimental Setup
Provides model configurations, evaluation metrics, and detailed result tables for the MRL experiments.
This section lists the hierarchical model dimensions used for each backbone, the evaluation metrics and datasets (Table 14), and the full result tables for both the baseline MRL and the proposed SpecMRL (Tables 15 and 16). The tables also include the $\tau$ compression levels and $\beta$ regularization settings explored in the experiments.
**Table 15.** Task-level breakdown of MTEB results in Table 1. Shaded rows denote SpecMRL, bold values indicate better overall results, parenthetical values report gains over MRL under each setup.
Extended Results
Extended tables confirm SpecMRL’s consistent gains across tasks and compression levels.
Table 17 reports per‑task MTEB scores when training is prolonged to 15 K steps. The authors note that SpecMRL’s advantage over baseline MRL persists throughout the longer schedule.
Table 18 compares SpecMRL with three existing MRL variants (MRL, ESE, MIPIC) across five compression levels ($\tau\!=\!1,2,4,8,16$). Across all tasks the spectral approach yields the highest average scores.