Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM

Sergii Kozyrev, Davyd Maiboroda

Quantizing the recurrent Gated DeltaNet layers of hybrid LLMs to 4-bit precision is safe and improves performance.

Why does Gated DeltaNet (GDN) maintain high accuracy under 4-bit quantization despite the common assumption that recurrent states accumulate quantization errors over long contexts?

Hybrid large language models use Gated DeltaNet (GDN) layers to summarize context, but community recipes protect these layers from 4-bit quantization, fearing that recurrent state errors will compound over long sequences. The authors demonstrate that this intuition is incorrect by building Minima, a model that quantizes all 496 linear layers to 4-bit weights and activations (NVFP4). The method relies on the architecture's own gating and delta-rule correction to naturally bound and erase quantization noise. Minima matches BF16 accuracy across reasoning and long-context benchmarks while reducing VRAM usage by 2.9× and increasing prefill throughput by 14–19%.

Paper Primer

The core mechanism hinges on three architectural properties: block-wise scaling localizes residual stream outliers, gate nonlinearities compress noise before it reaches control signals, and the delta-rule recurrence actively overwrites state errors as new tokens arrive. This creates a self-correcting system where the recurrent state error reaches a flat plateau rather than compounding over time.

Full 4-bit quantization of the GDN block does not degrade task accuracy.

Minima matches BF16 performance within seed noise across six accuracy suites, including AIME’25 and GPQA-Diamond, despite quantizing all 48 GDN layers. 5-task average difference of -0.52 points, well within the BF16 model's own seed spread.

Why did previous approaches protect the gate projections?

The community assumed that because the decay and write-strength gates control the recurrence, any quantization error in these parameters would compound rapidly over long contexts. The authors show these projections are actually the most robust because their log-space parameterization compresses pre-activation errors.

What is the role of the FP8 KV-cache scales?

While FP8 KV-cache reduces memory, it introduces a perplexity penalty that is 3× larger for quantized models than for BF16. Calibrated per-layer scales recover 83% of this penalty without impacting throughput.

For hybrid LLMs, the recurrent layers are the easiest to quantize, not the most fragile. Practitioners should quantize the entire backbone and use calibrated KV-cache scales to maintain long-context performance.

Introduction and Motivation

We expose why GDN’s recurrent state resists quantization error buildup and outline our contributions.

Hybrid large language models combine softmax attention with linear‑attention layers such as Gated DeltaNet (GDN), which use a recurrent state to summarize the entire context. Prior 4‑bit quantizations kept the GDN gates in higher precision, based on the error accumulation hypothesis that quantization errors in a recurrence would compound over long token sequences. This paper builds a fully 4‑bit model, Minima, and shows that GDN does not suffer from such error buildup while also providing a detailed mechanism study of why quantization works.

It updates a fixed‑size recurrent state each token by blending the previous state with a new contribution, using gates that control decay and write strength.

The belief that each quantized step in a recurrent update adds a small error, which over thousands of tokens grows into a large deviation that harms model quality.

The paper shows that recurrent states need not accumulate error, overturning a common quantization assumption.

Performance Benchmarks

Quantitative evaluation shows MINIMA matches BF16 accuracy while cutting memory and latency.

We benchmark MINIMA against BF16 and two community quantization recipes on a suite of language tasks and resource measurements.

MINIMA applies 4‑bit quantization to the Gated DeltaNet block, preserving the model’s representational capacity while shrinking weight storage.

How does MINIMA differ from the community 4‑bit recipes like Unsloth?

Unsloth quantizes only a small fraction of the model (mostly embeddings), leaving the GDN block at FP8. MINIMA, by contrast, quantizes the entire GDN recurrent state to 4 bits, which yields larger memory savings and faster prefill while still matching BF16 accuracy because the gates dampen quantization noise.

MINIMA quantization maintains accuracy comparable to BF16.

Serving-Stack Calibration

Background clarifies the serving‑stack calibration mismatch and related serving components.

The section surveys the serving pipeline elements that interact with Gated DeltaNet (GDN) and introduces the key mismatch that can silently corrupt gate activations.

A global‑scale calibration gap between per‑module NVFP4 scales and the fused serving GEMMs causes the GDN forget and write gates to receive mis‑scaled values, subtly degrading long‑context perplexity.

Other serving components—KV‑cache quantization, FP8 scaling, and the chat‑template harness—are described later, but their interaction with the calibration mismatch is essential for accurate long‑context evaluation.

Mechanism of Robustness

Analyzes the mechanisms that keep Gated DeltaNet robust under 4‑bit quantization.

We now turn to the core question: why does the Gated DeltaNet (GDN) remain stable when its internal state is quantized to 4 bits? The answer lies in three interacting mechanisms that together prevent error accumulation.

Impulse Decay is a gated exponential‑like decay where the gate parameters $\alpha$ and $\beta$ control how quickly a perturbation to the recurrent state is forgotten.

Step 0: s₁ = 0.9·0 + 0.1·1 = 0.1

Step 1: s₂ = 0.9·0.1 + 0.1·0 = 0.09

Step 2: s₃ = 0.9·0.09 = 0.081

Step 3: s₄ = 0.9·0.081 ≈ 0.073

After 10 steps the state is ≈ 0.035, already below 1/e ≈ 0.368 of the original impulse.

This toy illustrates how the gated decay rapidly shrinks an impulse, far faster than a pure $\alpha$‑only exponential (which would retain 0.9¹⁰ ≈ 0.35 after 10 steps).

How does Impulse Decay differ from a plain exponential decay that uses only a fixed $\alpha$ parameter?

In a plain exponential decay the state would evolve as sₜ₊₁ = $\alpha$·sₜ, so an impulse decays solely by repeated multiplication. Impulse Decay adds the $\beta$·$\Delta$ term, which injects a fresh correction each step, and the softplus‑parameterized $\alpha$ and $\beta$ keep the decay rate stable even when the underlying GEMM is quantized.

5.1 The inputs are not the reason. Table 2 shows that GDN layers receive inputs with extreme outliers (max/RMS ≈ 63.5, kurtosis ≈ 1.5 k) and many 1‑hot blocks, yet the per‑token activation error (A4) is uniformly 7.5–9.2 % across all roles.

The table presents statistics and error rates for various model components, including "GDN qkv/z/a/b", "GDN `out_proj`", "attention q/k/v", "attention `o_proj`", "MLP gate/up", and "MLP down". Columns include "max/RMS", "kurtosis", "1-hot blocks %", "A4 err %", and "W4 err %".

5.2 The protected projections are the safest ones. Table 3 reveals that quantizing the gate projections a and b shifts the final output y by only 2.1 % and 2.6 %, despite their own GEMM errors being as high as 11 %.

**Table 3.**

5.3 The recurrence bounds and erases the noise. Figure 1(a) shows a flat state error plateau over 32 K tokens, and Figure 1(b) demonstrates that a 1 % impulse injected at t = 1 024 decays to 1/e within a few hundred steps—far faster than the decay horizon implied by $\alpha$ alone.

**Figure 1.** State error of the FP32 lockstep recurrence over 32K tokens, five layers spread over depth (model layer indices). (a) With the full MINIMA quantization injected at every step, relS(t) plateaus immediately and stays flat — no accumulation. (b) A single 1% state perturbation at $t_0$=1,024 (log scale, 512-token bin means): the recurrence erases it within a few hundred to a few thousand steps, orders of magnitude faster than the decay-implied horizons of up to 62K tokens, because the delta rule overwrites the state along each new key.

5.4 End to end, context washes the error out. Figure 2 decomposes the quantization cost: weight cost declines toward the end of the window, while KV‑cache cost remains low, showing that the model’s internal dynamics continually suppress accumulated error.

**Figure 2.** The two quantization costs, decomposed by position in the 32K window (2K-token bins; full numbers in Table 4). Blue: weight cost (MINIMA–BF16 at matched KV precision) — largest at the start, falling to zero and below by the end of the window: the opposite of accumulation. Red: KV-cache cost (FP8–BF16 KV, same checkpoint) — small, rising with position, larger for MINIMA; this is the component the calibrated scales of §7 remove.

Overall, the combination of noisy‑tolerant gate projections, the Impulse Decay mechanism, and the recurrent write‑overwrites ensures that GDN’s state does not accumulate quantization error, contrary to the “error accumulation” hypothesis.

Quantization Gap Analysis

We quantify how quantization gaps evolve across context and fix measurement errors.

The Gated DeltaNet’s recurrent gates dampen past state information, preventing quantization error buildup over long contexts.

Minima’s weight‑quantization gap versus BF16 stays near zero early (+0.081 nats) and becomes negative (−0.053 nats) in the final 2 K tokens.

Per‑token NLL split by position shows +0.081 nats in the first half, +0.011 nats in the second half, and −0.053 nats at the tail.

FP8‑KV cost for Minima is roughly three times larger than BF16 and grows with position, indicating an attention‑path effect.

Measured cost rises across token positions and is ∼3× higher for Minima.

Synthesis: block scaling isolates residual outliers (§5.1), gate nonlinearities compress control signals (§5.2), the delta‑rule recurrence caps noise (§5.3), and end‑to‑end cost shrinks with context (§5.4).

Accurate measurement required repairing the pipeline; we identified four hidden corruptions affecting hybrid‑model NVFP4 deployments.

Fused‑GEMM scaling mismatches cause per‑module scales to differ by up to 2.75×, corrupting gate computations.

In every one of the 48 layers, qkv/z scales differ by 1.82× and b/a scales by 2.75×.

Repairing the fused scales raises the AIME score from 80.8 to 86.7, a +5.9 increase.

Post‑repair evaluation shows AIME = 86.7 versus the corrupted 80.8.

Long‑context perplexity drops from 10.84 to 6.86 after fixing the scale error, a reduction of −4.98.

Corrected model yields 6.86 at 32 K context versus the true 10.84.

GEMM‑level probe error falls to 0.002 after the scale correction.

Kernel‑vs‑reference error improves from 0.35/0.57 to 0.002.

Composite serving yields 10.04 perplexity versus 10.22 for text‑only BF16, a −0.18 advantage.

PPL@32 K = 10.04 (composite) vs. 10.22 (text‑only) for the BF16 model.

Raw‑completion harnesses are invalid for thinking models; lm‑eval’s local‑completions path bypasses the chat template, so “thinking disabled” never reaches the model.

Related Work

We situate our findings among prior linear‑attention, quantization, and KV‑cache work.

The 48 GDN layers carry no KV cache; only the 16 attention layers do. Storing their cache in FP8 (scale 1.0) is essentially free on tasks, and capacity grows 1.8–1.9×.

The only systematic cost appears at 32 K perplexity: +0.13 for BF16 and +0.41 for Minima, roughly three times larger, likely because Minima’s K/V projections are already W4A4, leaving less headroom before the cache rounds them again.

Calibrated scales close the gap. Minima+scales adds a static per‑tensor FP8 scale scheme (32 tensors on 16 attention layers) to an otherwise byte‑identical Minima recipe.

PPL@32 K drops from 10.84 to 10.50, recovering 83 % of the penalty; the residual +0.07 is below BF16’s own uncalibrated cost (+0.13). Throughput matches Minima within 0.4 % on every decode and prefill metric, making the scales performance‑free.

The practical recipe is unconditional: quantize everything, serve FP8 KV, and ship calibrated scales.

**Table 5.** Full throughput sweep (one RTX PRO 6000, TP=1, FP8 KV, same serving config as every accuracy number). Decode: 1024-token prompts, 1024 generated tokens, concurrency 1/8/32. Prefill: single request, one output token. MINIMA+scales differs from MINIMA only by the 32 KV scale tensors.

Linear attention and hybrids. Gated DeltaNet combines the parallelizable delta rule with Mamba2‑style gating; Qwen3.8‑27B deploys it as the dominant mixer in a hybrid stack.

Our results speak to the quantizability of this operator class, not to any single checkpoint’s training choices, because the mechanism rests on the operator’s own gating and correction structure.

Low‑bit LLM quantization. Weight‑only methods and weight‑activation methods established that outlier handling is the central difficulty of W4/W8 inference; block‑scaled microformats move the handling into the datatype, and NVFP4 is the hardware‑native instance we use.

Prior work targets transformer attention and MLPs; quantization of recurrent‑state mixers in large hybrids has, to our knowledge, not been studied—the public recipes for this model simply exempt them.

Concurrent with this work, QUASAR released a checkpoint of the same model that quantizes all 496 projections, GDN included, via quantization‑aware training—4‑bit weights are learned by distillation from the BF16 teacher.

It appeared after our measurement campaign closed; its model card reports a brief two‑task spot check but no controlled study or account of why the configuration survives. Our results show training is not necessary—calibration‑only PTQ reaches BF16‑level task accuracy.

Concurrent engineering work in our group extends the recipe studied here to the full model—embeddings, the language‑model head, and the multi‑token‑prediction head also at NVFP4 W4A4—and further to 3‑bit MLP codebooks with a distillation‑healed low‑rank residual. Notably, the sub‑4‑bit recipe keeps the GDN and attention projections sealed at 4‑bit weights, treating the result established here as its foundation.

That variant loads just 12.3 GiB of weights into VRAM—4.1× smaller than BF16’s 50.1 GiB, versus Minima’s 17.5 GiB—while sustaining over 1,200 output tokens/s at 128‑way concurrency on the same single‑GPU class; its accuracy evaluation uses different protocols and is outside this paper’s scope.

KV‑cache compression. Post‑training KV quantization targets the dominant memory consumer of long‑context attention serving. Hybrids shrink that consumer architecturally (here only 16 of 64 layers cache KV); our contribution is the interaction term—FP8 KV costs a W4A4 model 3× more perplexity than a BF16 model unless calibrated scales are shipped, after which the cost is below the BF16 model’s own.

Scope of evidence. One model family and size (Qwen3.8‑27B), one quantization format (NVFP4), evaluated to 32 K‑token perplexity and 64 K retrieval.

We did not run longer‑context stress tests: the mechanism study answers the accumulation question directly (state error flat over 32 K, gap shrinking with position, retrieval perfect at 64 K, ~14.5 K‑token generations matching BF16 token‑for‑token), and the bounded‑error mechanism predicts longer contexts, but 128 K+ behavior is extrapolation.

The concurrent QAT checkpoint of Counathe et al. is acknowledged but not benchmarked: it was released after our measurement campaign closed, and it belongs to a different recipe class (weights learned under quantization rather than rounded post hoc), so its scores are not comparable to numbers from our harness.

Minima+scales task scores are inherited from Minima rather than re‑measured, justified by the KV ablation showing no task movement for either model; PPL, RULER, and throughput were re‑measured.

Decode overhead. Minima trails RadixArk by 2–4 % on decode despite fewer weight bytes; profiling attributes this to small‑batch NVFP4 activation‑quantization overhead, a kernel‑level artifact rather than a property of the recipe.

Gates on other architectures. The gate‑shielding argument depends on the log‑space softplus/exponential parameterization; recurrent mixers with linearly‑parameterized decay may not enjoy it, and 0.1 % direct noise on $\alpha$ demonstrably harms the state.

We built a fully W4A4 NVFP4 version of a large hybrid LLM by post‑training quantization alone—all 496 backbone linear layers, GDN included—and found it matches BF16 across reasoning, knowledge, code, retrieval, and long‑context evaluation, while being the smallest and fastest‑prefill recipe in its cohort.

Technical Appendix

Appendix provides detailed numeric analyses, tables, and reproducibility notes for the study.

The instrumented FP32 recurrence (pure PyTorch) matches the fused chunkwise kernel on 32 K‑token inputs, yielding a median relative output difference of 7.7 × 10⁻⁴ and final‑state agreement of 4.4 × 10⁻⁴.

The BF16 kernel, limited by its rounding floor, shows a median error of 3.8 × 10⁻³ (≈2⁻⁸), which exceeds the smallest perturbation measured (10⁻³), so all error‑propagation experiments in §5.3 run on the clean FP32 reference.

Omitting the K‑query scaling factor introduces a spurious 91.2 % deviation (≈1 − 1/…), highlighting the scaling’s critical role in maintaining numerical stability.

Table 4 breaks down 32 K‑token WikiText‑2 NLL by 2 K‑token bins, showing that absolute NLL rises with position for all models, while the weight‑quantization gap (Minima − BF16) shrinks from +0.081 to +0.011 before turning negative, and the KV‑compression gap remains small but grows with position.

Table 5 reports a full throughput sweep on an RTX PRO 6000 (TP = 1, FP8 KV). Decode token rates range from 621 tok/s (BF16) to 1,174 tok/s (RadixArk) at batch size 32, while TPOT latency improves from 196 ms (BF16) to 19.6 ms (RadixArk) and TTFT drops from 27.3 s to 20.2 s for 32 K sequences.

Table 6 presents the full lockstep error‑propagation results (median over layers 0/14/30/46/62, 32 K tokens). Impulse decay per layer varies (1/e after 80–1,382 steps) and gate context averages $α=0.862$ with mean horizons $1/(1-α)$ ranging from 1,895 to 61,659 tokens; error percentages for variants such as $\alpha$‑noise 1 % reach up to 67.29 % max.

Minima quantizes 496 linear tensors to NVFP4 W4A4 across GDN and attention layers, while embeddings and norms remain in BF16. Seed statistics for Table 1 give 95 % confidence intervals of BF16 [79.2, 94.2] and Minima’s degenerate interval at 26/30, and truncation at the 32 K cap shows RadixArk’s higher cap‑hit rate (19.2 %) correlating with its AIME dip.

**Table 6.** Error propagation, full variant table. Table 6 gives the complete lockstep results behind §5.3 (median over layers 0/14/30/46/62, 32K tokens). Impulse decay per layer: 1/e after 550 / 164 / 281 / 80 / 1,382 steps; 1/10 after 2,937 / 2,233 / 2,270 / 2,239 / 2,693. Gate context: mean $\alpha$ = 0.862; per-layer mean horizons 1/(1 − $\alpha$) of 43,970 / 1,895 / 4,156 / 8,342 / 61,659 tokens; mean $\beta$ = 0.447.

**Table 6.** State error relS(t) vs. the clean FP32 recurrence (%, median over 5 layers) and resulting output errors. alpha/beta-noise rows apply synthetic multiplicative noise directly to the gates; impulse perturbs the state once at $t_0$=1,024 by 1%.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers