KronQ: LLM Quantization via Kronecker-Factored Hessian

Donghyun Lee, Yuhang Li, Ruokai Yin, Priyadarshini Panda

KRONQ improves LLM quantization by incorporating gradient covariance to account for output-side Hessian information.

How can we improve post-training quantization (PTQ) by accounting for output-side sensitivity in addition to input-side activations?

Existing post-training quantization methods like GPTQ rely solely on input activation statistics, implicitly assuming all output channels are equally important. This leads to significant performance degradation in ultra-low-bit regimes where output-side sensitivity is critical. KRONQ addresses this by using a Kronecker-factored Hessian approximation that incorporates gradient covariance. It applies bidirectional incoherence processing to suppress outliers on both input and output dimensions and uses joint Hessian traces to optimize bit-width allocation across sublayers. On LLaMA-3-70B at 2-bit weight-only quantization, KRONQ achieves 7.93 perplexity, whereas baseline methods diverge or produce degenerate results.

Paper Primer

The core move is the Kronecker-factored Hessian approximation $H \approx H_X \otimes H_G$. By capturing the gradient covariance $H_G$ alongside the input activation covariance $H_X$, the method accounts for output-side Hessian information that standard compensation-based methods ignore.

KRONQ is like a dual-axis stabilizer: it balances the weight matrix by applying orthogonal transforms to both the input and output dimensions, ensuring that rounding sensitivity is spread uniformly across all axes rather than concentrating on specific channels.

KRONQ enables stable 2-bit quantization where prior methods fail.

Comparison on LLaMA-3-70B (WikiText-2) shows KRONQ achieving 7.93 perplexity, while GPTQ and GPTAQ diverge (perplexity > 2000). Prevents total model collapse in ultra-low-bit regimes.

Joint Hessian sensitivity metrics improve mixed-precision allocation.

The sensitivity score $s_\ell = \operatorname{tr}(H_X^{(\ell)}) \cdot \operatorname{tr}(H_G^{(\ell)})$ differentiates sublayers (like Q, K, V) that share identical input statistics but differ in downstream gradient importance. Strictly better perplexity-bit-width trade-offs compared to activation-only metrics.

Why does this method require a backward pass when standard PTQ methods are "one-shot" and training-free?

KRONQ requires a single backward pass to compute the gradient covariance $H_G$. This is a one-time calibration cost that occurs before quantization, not a training or fine-tuning process, and it does not change the model's weights during inference.

Does the inclusion of $H_G$ increase the computational overhead during inference?

No. While $H_G$ is used during the calibration phase to guide the quantization process, it cancels out algebraically in the final weight update, meaning the quantized model has the same inference-time structure and latency as models quantized with GPTAQ.

Introduction and Motivation

The paper argues that PTQ must account for output‑side sensitivity, not just input activations.

Post‑training quantization (PTQ) is widely used to shrink large language models without any retraining, making deployment on limited hardware feasible.

Current PTQ pipelines, including second‑order methods such as GPTQ, build their quantization loss solely from input activation statistics, implicitly assuming that every output channel contributes equally to reconstruction error.

Quantization should minimize error that actually hurts the model, which requires looking at both how inputs vary ($H_X$) and how the loss surface reacts to weight changes ($H_G$).

The key limitation of input‑only quantization objectives is that they miss output‑side sensitivity, leading to suboptimal or even degenerate quantizations.

Related Work

We situate KRONQ among prior PTQ approaches, highlighting their reliance on input statistics and key distinctions.

Compensation‑based PTQ methods treat quantization error as a function of the input‑side activation covariance $H_X$, ignoring the output‑side gradient covariance $H_G$ that captures how sensitive each weight is to downstream loss.

GPTQ performs a column‑wise Optimal Brain Surgeon (OBS) update using only $H_X$, enabling fast one‑shot quantization of large language models.

GPTAQ augments GPTQ with a correction that compensates for input drift caused by quantization errors in earlier layers.

Extends GPTQ by incorporating attention‑aware Hessians that capture inter‑layer interactions within the attention module.

Reweights $H_X$ with end‑loss gradient saliency but still ignores cross‑output‑channel structure.

Adopts the Kronecker‑factored Hessian $H \approx H_X \otimes H_G$ and folds $H_G$ into a modified LDLQ solver via costly power‑iterated Hessian sketches.

Applies orthogonal rotations to weight matrices and activations to suppress outliers, enabling low‑bit weight‑and‑activation quantization.

Uses learned spin‑matrix transforms to rotate weights, achieving similar outlier suppression as QuaRot but with a learnable parameterization.

Establishes that incoherent weight and Hessian matrices yield lower quantization error, motivating orthogonal preprocessing.

Extends QuIP with randomized Hadamard transforms and lattice‑based vector quantization for practical 2‑bit quantization.

Intra‑layer mixed‑precision method that allocates bit‑widths within a layer using weight saliency.

Intra‑layer mixed‑precision approach that uses activation norms to guide bit‑width allocation.

Inter‑layer mixed‑precision method that ranks layers by Hessian trace $\operatorname{tr}(H)$.

Inter‑layer mixed‑precision method based on evolutionary search over bit‑width configurations.

Dynamic programming approach that selects per‑layer bit‑widths to satisfy a global memory budget.

Leverages a linearity theorem to allocate bits across layers, again using only $H_X$.

Mathematical Preliminaries

We establish the mathematical foundation for compensation-based quantization and incoherence processing in PTQ.

Post-training quantization (PTQ) aims to compress pre-trained models by approximating weights with lower-precision values. The core challenge is minimizing the reconstruction error of the layer outputs, which requires accounting for how weight changes propagate through the network.

Instead of quantizing weights in isolation, we treat the quantization of one column as a perturbation that must be compensated for by adjusting the remaining unquantized columns to keep the layer output stable.

While compensation methods optimize for reconstruction, they remain sensitive to the distribution of weight magnitudes. If weights are highly concentrated along specific axes, rounding errors become disproportionately large.

Incoherence processing spreads weight magnitudes uniformly across all dimensions by rotating the weight matrix and Hessian, ensuring that quantization sensitivity is not concentrated along any single coordinate axis.

Kronecker-Factored Quantization

We describe the Kronecker‑factored Hessian, bidirectional incoherence processing, and mixed‑precision allocation that together define the KRONQ method.

Standard PTQ treats the Hessian as purely input‑side, discarding how weight updates affect downstream activations.

The full second‑order curvature $H$ can be approximated as the outer product of an input‑side matrix $H_X$ and an output‑side matrix $H_G$, dramatically shrinking storage and computation.

Compute the Kronecker product $H_X\otimes H_G$, yielding a $4\times4$ matrix whose blocks are $2H_X$ and $5H_X$ placed on the diagonal.

Explicitly, $H_X\otimes H_G=\begin{bmatrix}8&2&4&1\\2&10&1&5\\4&1&12&3\\1&5&3&15\end{bmatrix}$.

Storing $H_X$ and $H_G$ requires $2^2+2^2=8$ numbers versus $4^2=16$ for the full $H$.

The factorization preserves the dominant curvature directions while halving memory, which is why KRONQ can afford to keep $H_G$ during PTQ.

Why isn’t the full Hessian $H$ stored directly if it contains more information?

Because $H$ scales as $d_{\text{out}}^2 d_{\text{in}}^2$, which is prohibitive for modern transformers; the Kronecker factorization reduces the cost to $d_{\text{out}}^2+d_{\text{in}}^2$ while retaining the most influential second‑order interactions.

Substituting the Kronecker approximation into the PTQ loss yields a tractable objective that still respects output‑side sensitivity.

Proposition 1 shows that, after applying the column‑wise OBS update to the augmented objective, the output‑side factor $H_G$ cancels analytically, leaving a simple closed‑form weight compensation.

We rotate and rescale both rows and columns of the weight matrix so that the input‑side and output‑side Hessians become nearly diagonal, eliminating harmful alignment with the coordinate axes.

Compute $S_X=\operatorname{diag}(4^{-1/4},9^{-1/4})\approx\operatorname{diag}(0.71,0.58)$ and $S_G=\operatorname{diag}(1^{-1/4},16^{-1/4})\approx\operatorname{diag}(1.00,0.50)$.

Rescale: $W' = S_G\,W\,S_X = \begin{bmatrix}1.42&-0.71\\0&0.87\end{bmatrix}$.

Apply a $2\times2$ Hadamard $U=V=\frac{1}{\sqrt{2}}\begin{bmatrix}1&1\\1&-1\end{bmatrix}$, yielding $W'' = U\,W'\,V^{\top}$, which is now dense and has no dominant diagonal.

The transformed $H_X$ and $H_G$ become $V\,H_X\,V^{\top}$ and $U\,H_G\,U^{\top}$, both of which are nearly diagonal with equalized entries.

Rescaling equalizes per‑row/column scales, while the Hadamard rotation removes any residual alignment, achieving the incoherence BiIP promises.

How does BiIP differ from the earlier input‑only incoherence methods?

Input‑only methods only rescale columns (via $S_X$) and rotate inputs; BiIP additionally rescales rows (via $S_G$) and rotates the output space, eliminating coherence on both sides of the weight matrix.

Bidirectional Incoherence Pre‑Processing (BiIP) – core steps.

Theorem 1 guarantees that the KRONQ loss (9) is unchanged by the BiIP transforms, so quantization can proceed on the incoherent weight matrix without revisiting the Hessians.

For any orthogonal $U\in\mathbb{R}^{d_{\text{out}}\times d_{\text{out}}}$ and $V\in\mathbb{R}^{d_{\text{in}}\times d_{\text{in}}}$, the KRONQ objective $\operatorname{tr}(H_G \Delta W H_X \Delta W^{\top})-\operatorname{tr}(W \Delta X X^{\top} \Delta W^{\top})$ is invariant under the transformation $W\!\leftarrow\!U W V^{\top}$, $H_X\!\leftarrow\!V H_X V^{\top}$, $H_G\!\leftarrow\!U H_G U^{\top}$.

We rank sublayers by the product of their input‑side and output‑side curvature traces; larger products indicate higher sensitivity to quantization error.

Figure 3 visualizes how the joint score reshuffles the sublayer ordering compared with the activation‑only score, directly translating into better perplexity‑bit‑width trade‑offs.

**Figure 1.** (a) Normalized diagonal entries of the gradient covariance $H_G$ for Q, K, V, and O projections in LLaMA-2-13B. Diagonal entries vary by orders of magnitude, revealing heterogeneous output-side sensitivity. (b) WikiText-2 perplexity of GPTQ, GPTAQ, and KRONQ on LLaMA-2-13B and LLaMA-3-8B across W4/W3/W2 weight-only quantization.

**Figure 2.** (a) $\mu$-incoherence of $\mathbf{H}_G$ before and after incoherence preprocessing across sublayers of LLaMA-2-7B. (b) Weight magnitude distribution of `Q_proj` under three configurations: original weights, after input-side incoherence ($\mathbf{H}_X$ only), and after bidirectional incoherence ($\mathbf{H}_X + \mathbf{H}_G$), where $CV_{in}$ and $CV_{out}$ denote the coefficient of variation of column and row norms, respectively.

**Figure 3.** Sublayer sensitivity rankings under the KronQ score $\text{tr}(\mathbf{H}_G) \cdot \text{tr}(\mathbf{H}_X)$ and the activation-only score $\text{tr}(\mathbf{H}_X)$ on LLaMA-2-7B and LLaMA-3-8B.

Experimental Results

KRONQ’s quantization outperforms prior PTQ methods across models and precisions.

Existing PTQ methods only use input‑side statistics, ignoring how weights react to output‑side curvature. This section shows how KRONQ leverages both sides to deliver superior quantization.

KRONQ attains the lowest WikiText‑2 perplexity across all evaluated models and bit‑widths, e.g., 7.61 PPL on LLaMA‑2‑7B at 2‑bit group quantization—a >35× reduction versus GPTQ.

Table 2 reports 7.61 PPL for KRONQ versus 274.0 PPL for GPTQ on the same setting.

5.1 Experimental Setup. We evaluate KRONQ on LLaMA‑2 (7B/13B/70B) and LLaMA‑3 (8B/70B), comparing against GPTQ and GPTAQ under three regimes: weight‑only (WxA16), group (WxA16, g=128), and weight‑and‑activation (WxA4) using the QuaRot framework for activations. All runs use 128 calibration samples from WikiText‑2 on A100 GPUs; HG is pre‑computed once per layer.

**Figure 5.** Inference efficiency of KronQ quantized models. (a) Peak VRAM and (b) decoding latency (TPOT) at batch size 1, relative to the bf16 baseline. For 70B, the bf16 baseline runs on two GPUs due to OOM on single GPU, while W4 runs on a single GPU.

5.2 Results on Uniform Precision. Table 1 shows KRONQ’s dominance for weight‑only quantization (W4/W3/W2) across perplexity and three zero‑shot reasoning benchmarks. Table 3 adds activation quantization (W2A4) where KRONQ cuts perplexity from 36.74 to 9.38 on LLaMA‑2‑7B, mirroring accuracy gains.

5.3 Results on Mixed‑Precision. Using the sensitivity score $s = \operatorname{tr}(H_G)\cdot\operatorname{tr}(H_X)$ (Eq. 14), we upgrade the most sensitive sublayers from W2 to W3. Table 6 demonstrates that the KRONQ joint score yields lower perplexity than the activation‑only score at identical average bit‑widths.

5.4 Memory and Latency. KRONQ reduces peak VRAM by 3.5–7.5× (W4/W2) and speeds decoding by 1.25–2.51× (TPOT) on 7B–13B models, making a 70B model fit on a single A100 at W4.

5.5 Analysis. Table 7’s ablations isolate each KRONQ component: replacing GPTAQ with GPTQ worsens perplexity; dropping diagonal rescaling in BiIP degrades performance; using only $H_G$ (output‑side) harms LLaMA‑2‑7B, confirming the complementary role of both $H_G$ and $H_X$.

Zero-shot Weight Quantization

Ablations reveal KRONQ’s advantage and the mixed impact of BoA across bit‑widths.

This ablation suite isolates the contribution of each design element by measuring zero‑shot reasoning accuracy after weight‑only quantization. The results expose where KRONQ’s Kronecker‑factored Hessian approximation pays off and how the optional BoA pre‑processing reshapes the trade‑off between bit‑width and stability.

The table presents a performance comparison of various quantization methods (GPTQ, GPTAQ, OmniQuant, BoA, and KRONQ) across different model architectures (LLaMA-2-13B, LLaMA-2-70B, LLaMA-3-8B, and LLaMA-3-70B) at different bit-widths (2, 3, and 4 bits). The metrics evaluated include PiQA, ArcE, ArcC, HS, WG, BoolQ, OBQA, and an average score (Avg↑).

KRONQ achieves the highest average zero‑shot score at 4‑bit across all evaluated models, surpassing the next‑best GPTAQ by +2.5 Avg.

4‑bit KRONQ avg = 68.7 versus GPTAQ avg = 66.2 (Table 10).

Adding the BoA component rescues KRONQ at 2‑bit, producing a valid average of 40.1 where plain KRONQ diverges (NaN).

KRONQ (BoA) 2‑bit avg = 40.1; KRONQ 2‑bit avg = NaN (Table 10).

BoA slightly degrades KRONQ performance at 3‑bit, dropping the average from 68.7 (4‑bit) to 67.3 (3‑bit with BoA), a –1.4 Avg loss.

KRONQ 4‑bit avg = 68.7; KRONQ (BoA) 3‑bit avg = 67.3 (Table 10).

Comparison with Gradient-Based Methods

Comparative analysis of gradient‑based PTQ methods across models and bit‑widths.

YAQA (Tseng et al., 2025) adopts the Kronecker‑factored Hessian form $H \approx H_X \otimes H_G$ but folds the gradient covariance $H_G$ into a modified LDLQ solver, whereas KronQ reserves $H_G$ solely for incoherence measurement and bit allocation.

The table compares the performance of two quantization methods, LNQ+GuidedQuant and KronQ, across different Llama 2 model sizes (7B, 13B, 70B) and bit-widths (W3 and W2).

**Table 11.** YAQA setup (LLaMA-3.1-8B-Instruct, WikiText-2, ctx 8192, INT4 per-channel).

Extending the comparison to Mistral‑7B (Jiang et al., 2024) confirms that KronQ’s advantage is not confined to the LLaMA family; it remains the lowest‑perplexity method across the evaluated bit‑widths.

**Table C.5** 3 and 4-bit Results on Group Quantization

Group quantization (block size $g=128$) further stresses the methods; here KronQ is compared against prior works such as OmniQuant and AWQ.

**Table 14.** Comparison with Previous Works on Group Quantization. Table 14 reports WikiText-2 perplexity under group quantization (g=128), where OmniQuant (Shao et al., 2023) and AWQ (Lin et al., 2024) results are taken from their papers.

Full W2/W3/W4 results (Table 15) reveal that KronQ not only leads on perplexity but also on downstream benchmarks (PiQA, ARC‑E/C, etc.), especially at 2‑bit where per‑channel schemes collapse.

**Table 15.** Group quantization (WxA16, g=128), full results (W2/W3/W4).

Weight-and-Activation Quantization

KRONQ delivers the best WikiText‑2 perplexity across all quantization settings.

We evaluate weight‑and‑activation quantization (WxA4) on WikiText‑2 using the QuaRot rotation framework for activation quantization.

KRONQ attains the lowest WikiText‑2 perplexity among all evaluated quantization methods.

Table 16 and Table 17 show KRONQ perplexities of 5.15 (LLaMA‑2‑7B, W4A4) and 5.83 (LLaMA‑3‑8B, W2A4), both lower than any competing method.

**Table 16.** Weight-and-activation quantization (WxA4) WikiText-2 perplexity.

**Table 17.** Weight-and-activation quantization (WxA4), W3A4/W4A4 results.

Theoretical Proofs

Formal proofs of KRONQ’s weight update, rotation invariance, and LDLQ optimality, plus algorithmic details.

The appendix supplies the missing theoretical guarantees for KRONQ: why the Kronecker‑factored Hessian cancels in the OBS weight update, why the objective is invariant to orthogonal rotations, and how the LDLQ factorisation yields a tight worst‑case proxy loss bound. It also lists the concrete algorithms used for BiIP preprocessing and the quantization loop.

Substituting the Kronecker‑factored approximation $H = H_X\otimes H_G$ yields $H^{-1}=H_X^{-1}\otimes H_G^{-1}$, and the two occurrences of $H^{-1}$ cancel algebraically, leaving a simple update that depends only on $H_X^{-1}$ and the error term.

Because the trace is cyclic and orthogonal matrices preserve inner products, the second term $\operatorname{tr}(H_G' W' (\Delta X X^{\top})' (\Delta W')^{\top})$ also remains unchanged, establishing full rotation invariance of the KRONQ loss.

Applying the incoherence bound to each factor separately yields an upper bound proportional to $\mu_X^{2}\mu_G^{2}\,k_X k_G\,\operatorname{tr}(H_X)\operatorname{tr}(H_G)/(4 d_{\text{in}} d_{\text{out}})$, which is strictly tighter than the QuIP bound when $H_G$ is low‑rank.

**Table 9.** Worst-case proxy loss under LDLQ rounding.

The table presents performance metrics for the LLaMA-2-7B model using various quantization methods (GPTQ, GPTAQ, KRONQ, OmniQuant, BoA) across different bit-widths (4, 3, 2). The metrics evaluated include PiQA, ArcE, ArcC, HS, WG, BoolQ, OBQA, and an average score (Avg↑).

Bidirectional Incoherence Pre‑Processing (BiIP)

KronQ quantization loop (per‑block processing)

The appendix proves that KRONQ’s Kronecker‑structured Hessian yields exact cancellation in the OBS update, remains invariant under orthogonal rotations, and leads to a provably tighter LDLQ loss bound, while the accompanying algorithms implement these guarantees in practice.

Implementation Details

Extended tables, figures, and practical notes on KronQ’s implementation.

Section C.8 reports zero‑shot evaluations of two 2025 models (DeepSeek‑R1‑Distill‑Llama‑8B and Gemma‑3‑12B‑IT) quantized to $W4$ with KronQ, GPTQ, and GPTAQ. All models are calibrated on 128 WikiText‑2 sequences without fine‑tuning. Metrics include log‑likelihood accuracy on MMLU and GPQA‑Diamond, exact‑match on AIME‑2024, and pass@1 on LiveCodeBench.

Across all benchmarks KronQ consistently yields lower perplexity or higher accuracy than the baselines, confirming the benefit of incorporating both input ($H_X$) and output ($H_G$) Hessian statistics. For example, on WikiText‑2 the KronQ‑quantized LLaMA‑3‑8B achieves a perplexity reduction of roughly $0.3$–$0.5$ points compared with GPTQ at the same average bit‑width.

**Figure.** Comparison of WikiText-2 Perplexity vs. Average Bit-width for LLaMA3-8B and LLaMA2-13B models using different quantization methods (SliM-LLM, SliM-LLM+, CMPQ, and KronQ).

Table 18 extends the mixed‑precision analysis of Section 5.3 to LLaMA‑3‑8B and LLaMA‑2‑13B. Using the joint trace score $tr(H_G)\cdot tr(H_X)$ yields strictly better perplexity‑bits trade‑offs than the activation‑only score $tr(H_X)$ for both architectures.

Tables 19–21 compare KronQ’s analytic allocation against three inter‑layer baselines (AMQ, Q‑Palette, HIGGS). With identical $3.1$‑bit budgets, KronQ attains lower perplexity (e.g., $6.52$ vs. $6.68$ for AMQ) while requiring only a single backward pass, eliminating the $5$–$79$ GPU‑hour searches required by the baselines.

**Figure a.** Weight distributions of LLaMA-2-70B

**Figure 6.** Weight magnitude distribution of the Q projection in the first layer under three configurations: original weights, after column-side incoherence ($H_X$ only), and after bidirectional incoherence ($H_X + H_G$). Top histograms show column norms; right histograms show row norms. $CV_{in}$ and $CV_{out}$ denote the coefficient of variation of column and row norms, respectively.

Table 22 reports per‑sublayer calibration memory (GiB) for LLaMA‑2‑7B, LLaMA‑3‑8B, and LLaMA‑2‑13B. KronQ initially stores the full $H_G$ matrix (e.g., $0.22$ GiB for $q_{proj}$) but releases it after BiIP, leaving a memory footprint identical to GPTAQ.

The primary limitation of KronQ is the extra backward pass needed to compute $H_G$, which adds offline compute and a transient peak memory overhead. During inference the method incurs only the orthogonal‑transform reversion cost $Θ(d_{in}\log d_{in}+d_{out}\log d_{out})$, comparable to the baseline per‑layer matrix‑vector multiply.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers