Normalized Low-Rank Adaptation
Jiale Kang, Ziyin Yue, Zheng Zhan, Yangyi Huang, Weiyang Liu
NoRA improves LoRA optimization by normalizing the down-projection matrix to eliminate scale imbalance.
How can we stabilize the training dynamics of LoRA by normalizing the low-rank projection?
Standard Low-Rank Adaptation (LoRA) often suffers from unstable training and suboptimal convergence because the random initialization of its down-projection matrix creates arbitrary scale imbalances across latent dimensions. Normalized Low-Rank Adaptation (NoRA) fixes this by normalizing the down-projection matrix along the rank dimension, ensuring each input coordinate is projected through a unit-norm latent direction. This simple constraint—or even just applying it at initialization—consistently accelerates convergence and improves downstream performance across pretraining, finetuning, and reinforcement learning without adding parameters or inference cost.
Paper Primer
LoRA parameterizes weight updates as the product of two low-rank matrices, but the early optimization phase is dominated by the random initialization of the down-projection matrix. NoRA treats this matrix as a set of projection vectors and enforces unit-norm constraints on them, effectively acting as a preconditioner that aligns the adapter's gradient flow with full finetuning.
The core mechanism is rank-dimension normalization: NoRA is the normalization of the down-projection matrix $A$ along the rank dimension $r$, while NoRA-init applies this normalization only once at the start of training. Think of this like a volume-leveler for a multi-channel audio mixer: it ensures that no single latent channel is disproportionately loud or quiet, preventing the "gradient collapse" often seen in standard LoRA.
NoRA significantly improves supervised finetuning performance over standard LoRA.
Across a suite of downstream tasks, NoRA increased the average performance score from 37.93 (LoRA) to 43.37.
NoRA-init captures the majority of NoRA's performance gains without requiring persistent normalization during training.
NoRA-init achieved an average score of 42.38, demonstrating that controlling the scale of the input-to-latent projection at initialization is the primary driver of optimization stability.
Why does this approach outperform spectral initialization methods like PiSSA in reinforcement learning?
Spectral methods rely on the singular-value decomposition of pretrained weights, which can become fragile during the unstable optimization regimes of reinforcement learning. NoRA avoids this dependency entirely, relying instead on a geometric constraint that remains robust across different training settings.
Does NoRA introduce any overhead during inference?
No. Because the normalization is applied to the down-projection matrix, the resulting transformation remains linear with respect to the input, allowing the normalized weights to be merged into the pretrained model exactly like standard LoRA.
The paper identifies the down-projection matrix as a critical, under-explored design dimension in PEFT. By showing that LoRA is essentially full finetuning under an implicit input-side preconditioner, the authors provide a theoretical basis for why simple normalization leads to such consistent empirical gains.
Researchers and practitioners should treat the down-projection matrix as a primary target for initialization and regularization, as rank-dimension normalization provides a "free" performance boost that is now a drop-in replacement for standard LoRA.
Motivation and Problem Framing
LoRA’s zero‑initialized up‑projection makes early training unstable, motivating a normalization fix.
Parameter‑efficient fine‑tuning is essential as large language models grow, and LoRA has become a de‑facto standard because it reduces trainable parameters while preserving performance. Yet the paper notes that LoRA’s training dynamics remain poorly understood, especially the effect of its initialization scheme.
LoRA sets the up‑projection matrix $B$ to zero at the start, so the early forward and backward passes are governed solely by the randomly initialized down‑projection $A$, which can produce wildly varying feature scales and unstable gradients.
Given this observation, the authors ask whether explicitly regularizing the down‑projection matrix $A$ can improve LoRA’s optimization behavior.
The core problem is that LoRA’s zero‑initialized up‑projection creates early training instability.
LoRA Foundations
Formalizes LoRA’s low‑rank update and frames it as a latent‑feature projection.
We formalize LoRA’s low‑rank weight update and recast it as a latent‑feature projection.
LoRA injects a small trainable bottleneck into a frozen pretrained weight matrix by expressing the update as a low‑rank product.
The NoRA Mechanism
Normalize LoRA’s down‑projection to keep column scales unit, stabilizing early training.
Standard LoRA’s zero‑init up‑projection yields tiny early gradients, making the optimizer wobble before it finds a useful direction. Normalizing the down‑projection eliminates that wobble by fixing the scale of each input coordinate.
Instead of normalizing the latent feature after it is projected, we normalize the LoRA down‑projection matrix itself so every column has unit norm; this removes column‑wise scale imbalance and keeps the forward pass linear in the input.
Compute column norms: ‖a₁‖₂=√(2²+1²)=√5≈2.24, ‖a₂‖₂=√(0.5²+ (‑2)²)=√4.25≈2.06, ‖a₃‖₂=√((‑1)²+3²)=√10≈3.16.
Normalize columns: a₁′=[2/2.24, 1/2.24]≈[0.894, 0.447], a₂′=[0.5/2.06, ‑2/2.06]≈[0.243, ‑0.971], a₃′=[‑1/3.16, 3/3.16]≈[‑0.316, 0.949].
Form Norm(A) by stacking the normalized columns a₁′, a₂′, a₃′.
Compute Norm(A) x = a₁′·1 + a₂′·(‑1) + a₃′·2 ≈ [0.894 − 0.243 + (‑0.632), 0.447 + 0.971 + 1.898]ᵀ ≈ [0.019, 3.316]ᵀ.
Assume B = [[1, 0],[0, 1]] (identity). Then $\Delta$y = $\alpha$ B (Norm(A) x) = [0.019, 3.316]ᵀ.
Unit‑norm columns guarantee that the magnitude of Norm(A) x is driven solely by the input x, not by arbitrary scaling in A; this is why early gradients match those of full fine‑tuning.
How does NoRA differ from the standard LoRA update $\Delta$y=$\alpha$ B A x?
Standard LoRA leaves the columns of A with whatever random scale they receive at initialization, so each input coordinate gets a different effective learning rate. NoRA first normalizes each column of A to unit length, making all coordinates start with the same learning rate and keeping the forward pass linear, which stabilizes early gradient dynamics.
At the start of LoRA training the gradient magnitude seen by the adapter is proportional to the squared norms of A’s columns; normalizing those columns makes the effective preconditioner close to the identity matrix, so gradients have the same scale as in full fine‑tuning.
Why do column norms of A act like per‑coordinate learning rates?
Because the LoRA update $\Delta$W=$\alpha$ B A x can be rewritten as $\Delta$W=−$\eta$ G ($\alpha$² AᵀA). The diagonal of $\alpha$² AᵀA is $\alpha$²‖$a_j$‖₂², which multiplies the gradient for each input coordinate j. If a column is larger, its coordinate receives a proportionally larger step, effectively scaling its learning rate.
**Figure 1.** Illustration and optimization behavior NoRA. **Left:** Each input-to-latent projection vector is normalized along the LoRA rank dimension. **Right:** Gradient-norm dynamics of different initialization methods and LoRA ranks on LLaMA-3.2-3B trained on the Math dataset.
**Figure 2.** Training dynamics comparison of full finetuning, LoRA, PiSSA, and NoRA on the SFT task.
Pretraining Performance
NoRA‑init boosts downstream accuracy and speeds up convergence during pretraining.
NoRA‑init improves average downstream accuracy over the standard low‑rank parameterization.
Table 4 shows NoRA‑init achieving a 1.1 % higher average accuracy than the standard low‑rank baseline.
**Table 4.** Zero-shot performance of 340M models trained on SlimPajama (Soboleva et al., 2023). Commonsense reasoning tasks are evaluated with lm-evaluation-harness (Gao et al., 2024); the recall-intensive task follows prefix-linear-attention (Arora et al., 2024) with 2K input tokens. “-” indicates a collapse in perplexity.
The ablation study in Section 4.1 shows that column‑wise normalization (Norm$_k$) yields negligible gains, whereas row‑wise normalization (Norm$_r$) consistently improves performance across random and deterministic initializations.
Finetuning and RL Results
Finetuning and RL ablations reveal how normalization drives performance gains.
Standard LoRA zero‑initializes its up‑projection, which makes early‑training gradients unstable. NoRA normalizes the latent features before the up‑projection, keeping the scale consistent throughout training.
**Table 5.** Comparison of PEFT methods under supervised finetuning and retained benchmark settings.
NoRA achieves the best overall supervised‑finetuning performance, raising the average to 43.37.
Improves the average from 37.93 with standard LoRA to 43.37, a gain of 5.44 points (Table 5).
NoRA‑init, which normalizes only at initialization, captures most of the benefit, lifting the average to 42.38.
Improves the SFT average from 37.93 to 42.38 (Table 5), a gain of 4.45 points.
**Table 6.** Comparison of accuracy and pass rate on the RLVR task. All numbers are reported in percentages.
Under RLVR, NoRA raises the overall average to 44.4, outperforming LoRA by 1.6 points.
Table 6 shows NoRA at 44.4 % versus LoRA at 42.8 % and the base model at 41.0 %.
Related Work and Implementation Details
Survey of PEFT methods and key experimental settings.
This appendix collates prior PEFT work, presents the experimental configurations used throughout the paper, and highlights the broader implications of our proposed NoRA method.
PEFT adapts a large pretrained model by inserting a tiny set of trainable modules while keeping the original weights frozen, so only a few extra parameters are updated during downstream training.
**Table 1.** Overview of representative PEFT methods categorized by their primary mechanism, including forward computation, parameterization and initialization strategy.
Prior work has explored many PEFT variants—LoRA for its simplicity, DoRA for weight‑decomposed updates, PiSSA for singular‑value‑based initialization, and others for quantization or adaptive rank allocation—yet most focus on improving capacity or efficiency rather than the initialization geometry.
**Table 2.** Overview of all the training settings in our experiments.
Pretraining experiments follow a fixed schedule: 20 480 steps on FineWeb‑10BT with a sequence length of 2 048, a global batch size of 256, and AdamW (peak LR = 3 × 10⁻⁴, $\epsilon$ = 10⁻¹⁵) warmed up for 1 024 steps then cosine‑decayed to 10 % of the peak.
**Table 7.** Hyperparameter settings for the pretraining experiments.
Supervised finetuning (SFT) adopts a rank of 32 for all methods, sets $\alpha$ = r for NoRA, and uses a cosine‑decay learning rate of 2 e‑5 with batch size 128; dropout is fixed at 0.0 across the board.
**Table 8.** Hyperparameter settings for the Llama-3.2-3B supervised finetuning experiments.
Reinforcement‑learning experiments use the DeepSeek‑R1‑Distill‑Qwen‑1.5B model on the DAPO‑Math‑17K dataset, training for 1 024 steps with a batch size of 128, learning rate 1 × 10⁻⁵, and eight sampled responses per prompt.
This table lists the hyperparameters used for the model training process.
In summary, NoRA normalizes the down‑projection along the rank dimension, correcting the scale imbalance of the implicit low‑rank preconditioner and consistently improving convergence, stability, and downstream performance across all three training regimes.
Questions & answers
What is the main contribution of the NoRA paper?
NoRA introduces rank-dimension normalization of LoRA's down-projection matrix A, ensuring each column has unit norm so that all input coordinates start with the same effective learning rate. The paper also proposes NoRA-init, a lighter variant that applies this normalization only once at initialization rather than throughout training.
What problem does NoRA address in standard LoRA?
Standard LoRA's random initialization of the down-projection matrix A creates arbitrary scale imbalances across latent dimensions, causing unstable early training and suboptimal convergence. Because column norms of A act as per-coordinate learning rate multipliers, unequal norms mean different input coordinates receive disproportionately large or small gradient steps.
Why does LoRA's initialization cause training instability?
The LoRA weight update can be rewritten as ΔW = −η G (α² AᵀA), where the diagonal of α² AᵀA equals α²‖aⱼ‖₂² for each input coordinate j. If columns of A have unequal norms, each coordinate receives a different effective learning rate, causing the optimizer to wobble before finding a useful direction.
How does NoRA's normalization mechanism work?
NoRA normalizes each column of the down-projection matrix A to unit length along the rank dimension r before the forward pass, so every input coordinate is projected through a unit-norm latent direction. Because the transformation remains linear with respect to the input, this acts as a preconditioner that aligns gradient flow with full finetuning without changing the model's functional form.
What is the difference between NoRA and NoRA-init?
NoRA applies rank-dimension normalization of the down-projection matrix A continuously throughout training, while NoRA-init applies this normalization only once at the start of training. The paper shows that even the one-time initialization variant consistently improves convergence and downstream performance.
Does NoRA add parameters or inference overhead?
No. Because the normalization is applied to the down-projection matrix and the resulting transformation remains linear with respect to the input, the normalized weights can be merged into the pretrained model exactly like standard LoRA, incurring no additional inference cost.
What datasets and benchmarks were used to evaluate NoRA?
Pretraining experiments used FineWeb-10BT with 20,480 steps and sequence length 2,048. Supervised finetuning (SFT) experiments used rank 32 with α = r and a cosine-decay learning rate of 2e-5. Reinforcement learning experiments used the DeepSeek-R1-Distill-Qwen-1.5B model on the DAPO-Math-17K dataset for 1,024 steps.
What were the key experimental settings for pretraining?
Pretraining used a fixed schedule of 20,480 steps on FineWeb-10BT, a sequence length of 2,048, a global batch size of 256, and AdamW with a peak learning rate of 3×10⁻⁴ and ε = 10⁻¹⁵, warmed up for 1,024 steps then cosine-decayed to 10% of the peak.
What were the key experimental settings for reinforcement learning?
RL experiments used the DeepSeek-R1-Distill-Qwen-1.5B model on DAPO-Math-17K, training for 1,024 steps with a batch size of 128, a learning rate of 1×10⁻⁵, and eight sampled responses per prompt.
What do the ablation results show about the direction of normalization?
The ablation study in Section 4.1 shows that column-wise normalization (Normₖ) yields negligible gains, whereas row-wise normalization (Normᵣ), i.e., normalization along the rank dimension, consistently improves performance across both random and deterministic initializations.
Why does NoRA outperform spectral initialization methods like PiSSA in reinforcement learning?
Spectral methods such as PiSSA rely on singular-value decomposition of pretrained weights, which can become fragile during the unstable optimization regimes of reinforcement learning. NoRA instead relies on a geometric unit-norm constraint that does not depend on SVD and remains robust across different training settings.
How does NoRA compare to related PEFT methods such as DoRA and PiSSA?
Prior PEFT methods like DoRA (weight-decomposed updates) and PiSSA (singular-value-based initialization) focus primarily on improving capacity or efficiency rather than the initialization geometry of the down-projection matrix. NoRA specifically targets the scale imbalance introduced by random initialization of A, providing a complementary and drop-in improvement over standard LoRA.
What is the theoretical framing the paper provides for NoRA's effectiveness?
The paper shows that the LoRA update is equivalent to full finetuning under an implicit input-side preconditioner defined by α² AᵀA, and that normalizing A to unit-norm columns corrects the scale imbalance in this preconditioner. This theoretical framing explains why rank-dimension normalization consistently improves gradient flow and convergence.
Across which training regimes does NoRA demonstrate improvements?
NoRA demonstrates consistent improvements across all three training regimes evaluated in the paper: pretraining, supervised finetuning (SFT), and reinforcement learning (RL), without adding parameters or inference cost.
What practical recommendation does the paper make for practitioners?
The paper recommends that researchers and practitioners treat the down-projection matrix as a primary target for initialization and regularization, and adopt rank-dimension normalization as a drop-in replacement for standard LoRA to obtain a 'free' performance boost.
What model is used in the reinforcement learning experiments?
The RL experiments use the DeepSeek-R1-Distill-Qwen-1.5B model trained on the DAPO-Math-17K dataset.
Does the paper report specific numerical performance improvements (e.g., accuracy or loss numbers)?
The paper does not provide specific numerical performance improvement figures (such as exact accuracy or loss values) in the content provided; it describes consistent improvements in convergence and downstream performance across all three training regimes.
Who are the authors of the NoRA paper and where was it published?
The paper does not specify the authors' names in the provided content. It is available on arXiv at arxiv.org/abs/2608.31036, but the paper does not state a venue or publication date beyond the arXiv identifier.
Key terms
- LoRA (Low-Rank Adaptation)
- A parameter-efficient fine-tuning method that parameterizes weight updates as the product of two low-rank matrices, reducing the number of trainable parameters while preserving model performance.
- NoRA (Normalized Low-Rank Adaptation)
- A variant of LoRA that normalizes the down-projection matrix along the rank dimension so each column has unit norm, correcting scale imbalances and stabilizing training.
- NoRA-init
- A lighter variant of NoRA that applies rank-dimension normalization to the down-projection matrix only once at the start of training rather than continuously throughout.
- down-projection matrix (A)
- The first of the two low-rank matrices in LoRA, which projects the input from the original high-dimensional space down to a lower-rank latent space.
- up-projection matrix (B)
- The second of the two low-rank matrices in LoRA, which projects the latent representation back up to the original high-dimensional output space; it is initialized to zero in standard LoRA.
- rank dimension (r)
- The bottleneck dimensionality of the low-rank matrices in LoRA, controlling the number of latent directions used to represent the weight update.
- rank-dimension normalization
- The operation of normalizing each column of the down-projection matrix A to unit length along the rank dimension r, ensuring all latent directions have equal scale.
- preconditioner
- A matrix applied to gradients during optimization to rescale or rotate them, improving the conditioning of the optimization problem and accelerating convergence.
- PEFT (Parameter-Efficient Fine-Tuning)
- A family of techniques for adapting large pretrained models to new tasks by training only a small subset of parameters rather than the full model.
- PiSSA
- A LoRA initialization method that uses the singular-value decomposition (SVD) of pretrained weights to initialize the low-rank adapter matrices.
- DoRA (Weight-Decomposed Low-Rank Adaptation)
- A PEFT variant that decomposes weight updates into magnitude and direction components to improve adaptation capacity.
- singular-value decomposition (SVD)
- A matrix factorization technique that decomposes a matrix into orthogonal directions and their associated magnitudes (singular values), used by spectral initialization methods like PiSSA.
- FineWeb-10BT
- A large-scale web text dataset used in the paper's pretraining experiments, containing approximately 10 billion tokens.
- DAPO-Math-17K
- A mathematical reasoning dataset containing approximately 17,000 examples used in the paper's reinforcement learning experiments.
- AdamW
- A widely used adaptive gradient optimizer that combines the Adam update rule with decoupled weight decay regularization.
- cosine decay
- A learning rate schedule that reduces the learning rate following a cosine curve from its peak value down to a specified minimum over the course of training.
- per-coordinate learning rate
- An effective learning rate that differs for each input dimension, arising here because the column norms of A scale the gradient update for each corresponding input coordinate.
- gradient collapse
- A training failure mode in which gradients become vanishingly small for certain dimensions, preventing those parameters from being updated effectively.