On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability

Zihan Qiu, Zekun Wang, Xiao Li, Yanpeng Li, Yang Xu, Yixuan Wang, Huaqing Zhang, Rui Men, Bochao Mao, Chengruidong Zhang, Fan Zhou, Hao Luo, Haofeng Huang, Haoran Lian, Haoyan Huang, Hongqing Chen, Jianwei Zhang, Jing Xu, Junjie Wang, Langshi Chen, Liangyu Wang, Linlang Jiang, Man Yuan, Minmin Sun, Peng Jin, Siqi Zhang, Siyu Wang, Xingzhang Ren, Yakai Wang, Yi Zhang, Yiming Dong, Yizhong Cao, Yubo Ma, Yunfei Mao, Bo Zheng, Dayiheng Liu

Qwen3.8-Flash-Next optimizes sparse MoE efficiency through hybrid token mixing, gated residual streams, and sparse attention.

How does the Qwen3.8-Flash-Next architecture combine sparse mixture-of-experts, gated residual connections, and n-gram embeddings to maintain high performance while reducing active parameter count?

Large-scale sparse models struggle to balance long-context retrieval performance with the high compute and memory costs of dense attention and standard residual connections. The authors introduce a hybrid architecture that combines Gated DeltaNet (GDN) for linear-cost recurrence with Qwen Sparse Attention (QSA) for long-context indexing, while widening the residual stream into a Gated Residual (GR) structure. This design achieves the performance of a 397B-parameter predecessor while using only one-third the activated parameters and one-ninth the training FLOPs.

Paper Primer

The architecture addresses the quadratic cost of global attention by using a layer-wise hybrid: recurrent GDN layers compress prefixes into fixed-size states, while periodic full-attention layers (or QSA at continued-pretraining) handle direct token retrieval. The Gated Residual (GR) mechanism widens the residual stream into four parallel branches, using a data-dependent elementwise gate to decide how information is read and written across these branches.

QSA significantly accelerates long-context inference without degrading general task performance.

At 1M context length, QSA provides a 7.6× speedup in prefill and 4.9× in decode compared to dense attention, while matching or exceeding the baseline on seven of eight benchmarks.

The Gated Residual (GR) design improves both training stability and downstream accuracy.

GR eliminates the need for explicit gradient clipping and maintains stability at 4× the optimal learning rate, where previous architectures spiked.

Why use a hybrid of recurrent layers and sparse attention instead of just one?

Recurrent layers like GDN provide linear-cost compression of prefixes, but they struggle to replicate the exact token-level retrieval of full attention. Interleaving sparse attention layers allows the model to retain direct content access while keeping the overall computational cost manageable.

How does the Gated Residual (GR) differ from standard residual connections?

Standard residuals use a single path that attenuates signals over depth. GR widens this to four parallel branches and uses a data-dependent elementwise gate to dynamically route information, allowing the model to preserve early-layer features while keeping local processing efficient.

Architectural efficiency is a joint problem of loss, benchmarks, and stability; this paper demonstrates that widening residual capacity and using hybrid token-mixing allows for massive compute savings without sacrificing the capabilities of larger, denser models.

Introduction and Motivation

We expose the efficiency bottleneck and motivate sparse‑expert design to boost capability with less compute.

Large‑scale language models still spend most of their compute on dense attention, limiting scalability. Qwen3.8‑Flash‑Next targets this bottleneck by drastically reducing the number of active parameters per token while preserving quality, using sparse attention and a gated residual pathway.

The paper aims to balance four intertwined goals—model loss, downstream performance, computational efficiency, and training stability—by redesigning attention and residual connections.

Shifting from dense to sparse architectures is the key lever for achieving higher efficiency without sacrificing capability.

Architecture Overview

How the model mixes tokens efficiently by blending recurrent compression with periodic global attention.

Full self‑attention gives every token direct access to all previous tokens, but its cost grows quadratically with sequence length and its KV cache grows linearly during generation. The hybrid design replaces most layers with a recurrent compressor (GDN) and inserts a full‑attention layer every fourth block to retain global access.

The model compresses the token prefix into a compact recurrent state with Gated DeltaNet, then periodically injects a full‑attention layer that can read any token directly, achieving both efficiency and long‑range fidelity.

Decay: $eS_0 = 0.8 \times S_0 = \begin{bmatrix}0&0\\0&0\end{bmatrix}$.

Residual: $e_1 = v_1 - (eS_0)^\top k_1 = (2,1) - (0,0) = (2,1)$.

Update: $S_1 = eS_0 + 0.5\,k_1 e_1^\top = 0.5 \begin{bmatrix}0\\1\end{bmatrix}\begin{bmatrix}2&1\end{bmatrix} = \begin{bmatrix}0&0\\1&0.5\end{bmatrix}$.

Output: $y_1 = S_1^\top q_1 = \begin{bmatrix}0\\1\end{bmatrix}$.

At $t=2$ use $q_2=(0,1)$, $k_2=(1,0)$, $v_2=(1,3)$, $\alpha_2=0.7$, $\beta_2=0.6$ and repeat the same four operations, yielding $S_2$ and $y_2$.

The recurrence stores a low‑rank summary of past keys and values; repeated keys only modify the existing association instead of creating new outer‑product terms.

How does the GDN hybrid differ from plain linear attention?

Linear attention simply adds outer‑products $k_t v_t^\top$ to a cumulative matrix, which grows without bound and treats each key independently. GDN first decays the old matrix, then writes only the residual error, so repeated or similar keys update an existing association instead of creating redundant terms. This targeted erase‑and‑write keeps the state compact and stable.

Standard dense attention that computes a softmax over all pairwise query–key dot products, giving each token direct access to every other token.

Why keep a full‑attention layer at all if GDN is much cheaper?

GDN compresses history into a fixed‑size state, which inevitably loses some fine‑grained token‑level information. The periodic full‑attention layer restores exact token‑level access, ensuring that rare or long‑range patterns can still be captured directly.

**Figure 1.** **Qwen3.8-Flash-Next architecture.** Token mixing alternates three GDN layers with one QSA layer per block of four. Every sublayer reads and writes through GR, which widens the residual stream and gates the read elementwise. An n-gram embedding layer at Layer 2 scales capacity off the accelerator via host-memory prefetching. The MTP module reuses QSA indices across speculative decoding steps.

**Figure 2.** The Gated DeltaNet token mixer. The projected query, key, and value streams pass through short causal convolutions; queries and keys are L2-normalized before the gated delta recurrence. The decay gate $\alpha_t$ and write gate $\beta_t$ control the recurrent update, while a sigmoid output gate modulates the zero-centered RMS-normalized output.

Qwen Sparse Attention

Qwen Sparse Attention replaces dense attention with a block‑wise top‑k selection to cut compute on long sequences.

Sparse attention tackles the quadratic scaling of softmax attention by limiting each query to a small set of context blocks. Existing indexers still incur $O(n^2)$ cost, which becomes a bottleneck for very long sequences.

QSA first compresses the input sequence into micro‑blocks, scores each block, and then lets each query attend only to the top‑k blocks, turning the dense $O(n^2)$ attention into roughly $O(n\,k)$ work.

Project each token: compute queries $Q_i$ and keys $k_i$ via the linear layers.

Group keys into 2‑token blocks and average‑pool: block 0 gets $\frac{k_0+k_1}{2}$, block 1 gets $\frac{k_2+k_3}{2}$, etc., then apply RMSNorm to obtain compressed keys $k_b$.

For each query token $i$, compute block‑level scores $I_{i b}$ by summing ReLU‑activated dot‑products across the two heads, respecting the block‑causal mask (e.g., token 5 can only score blocks 0–2).

Select the top $K_B=2$ blocks per query (e.g., blocks 1 and 2 for token 5) and expand them back to their original token indices (4 tokens total).

Combine these selected tokens with any remaining tokens in the incomplete final block to form the final attention set (here, 4 tokens instead of 8).

This toy example shows that QSA cuts the number of attended tokens roughly in half while still respecting causality, illustrating the core efficiency gain.

**Figure 3.** Overview of Qwen Sparse Attention (QSA). The QSA indexer (left) uses a compressed causal attention mask to score key blocks and select the top-k indices. These indices are expanded into a micro-block sparse attention mask for sparse core attention (right).

QSA indexer and top‑k block selection.

QSA Training and Performance

QSA matches full attention on benchmarks while boosting long‑context efficiency.

QSA raises the average benchmark score by +0.9 points over the full‑attention baseline.

Table 2 reports 76.8 vs 75.9 average across eight benchmarks.

**Figure 4.** Training LM loss with and without QSA. Curves are smoothed with a 200-step moving average. The shaded region marks the final stage of continued pretraining, and the inset shows the per-step loss difference between QSA and the full-attention baseline in this region.

QSA Ablations and Efficiency

Ablations reveal how QSA and its design choices affect performance and efficiency.

We probe how each component of QSA and its surrounding architecture influences both accuracy and latency.

QSA raises the RULER score from 90.08 to 93.00 beyond 512 K tokens.

Table 3 reports the scores at long sequence lengths.

On the MRCR benchmark at 512 K tokens, QSA lifts the score from 30.66 to 40.53.

Table 3 shows the MRCR results.

At 1 M tokens, QSA improves MRCR from 20.71 to 26.44.

Table 3 reports the 1 M results.

Four‑step speculative decoding with QSA yields an average accepted length of 4.07 versus 4.06 for full attention.

Table 4 compares the mean accepted length.

Reusing the top‑k indices across speculative decoding steps does not noticeably affect the mean accepted length, confirming that index reuse is safe.

QSA retains performance with as few as four indexer query heads, balancing speed and accuracy.

**Figure 5.** Architecture ablations of QSA on RULER. (a) QSA performance with different micro-block sizes; "Keep x" indicates the number of IndexShare indexer layers retained for computation. (b) Performance with different numbers of indexer query heads after dense distillation and sparse training.

**Figure 6.** Kernel-level latency of QSA across context lengths during prefill and decode. Panels (a,b) compare indexer latency under different compression ratios, while panels (c,d) compare kernel-level attention latency between dense GQA and QSA, including both the indexer and sparse core attention. Chunked prefill uses a 16K-token chunk with batch size 1; decode uses batch size 4 and `next_n`= 4, corresponding to three MTP prediction steps. Arrows indicate speedups at a context length of 1M.

Gated Residual Connections

Introduce a per‑channel gate after RMSNorm to stabilize deep residual streams.

Training deep transformer layers can become unstable because residual connections pass unfiltered activations, leading to exploding or vanishing gradients. A simple fix is to insert an elementwise gate after RMSNorm, which we call Gated Residual (GR).

GR adds a data‑dependent sigmoid gate after each RMSNorm branch, letting the network modulate how much of the normalized signal flows forward, which stabilizes training.

Stack the branches: vec($b_R$) = [1,2,1,2,1,2,1,2, 2,1,2,1,2,1,2,1] (length 16).

Apply $W_d$ (1 × 16) → scalar $a$ = $\Sigma$ vec($b_R$) = 24; SiLU(a) ≈ 24 (since a > 0).

Apply $W_u$ (16 × 1) → vector $g$ = 24 · 1 (broadcast) = [24,…,24]; sigmoid(g) ≈ 1 for each entry, so G ≈ all‑ones.

Gate each branch: G₁ ⊙ $b_{R₁}$ = $b_{R₁}$, G₂ ⊙ $b_{R₂}$ = $b_{R₂}$; average gives x = ($b_{R₁}$+$b_{R₂}$)/2 = [1.5,1.5,…,1.5].

Compute write scalar: s = 2 $\sigma$($W_w$ vec($b_R$)) → assume $W_w$ vec($b_R$) = 0, so $\sigma$(0)=0.5, thus s = [1,1].

Update branches: R'_i = $R_i$ + $s_i$ y, where y = F(x) (let y = [0.1,…,0.1] for demo). Result: R'_1 = R₁ + 0.1, R'_2 = R₂ + 0.1.

The example shows that the gate can fully pass or fully suppress each channel, while the scalar $s$ ensures both branches receive the same amount of new information, preventing one branch from overwhelming the other.

How does Gated Residual differ from a standard residual connection with a simple scalar gate?

Standard residual adds the block output directly (y) to the input without any modulation, and a scalar gate would apply the same factor to every channel of every branch. GR first normalizes each branch, computes a channel‑wise gate that depends on the concatenated branches, and uses a separate per‑branch scalar $s$, so the gating is both data‑dependent and fine‑grained.

Evaluation and Efficiency

We assess loss‑accuracy trade‑offs and compute gains of the sparse and gated design.

Recall that Qwen3.8‑Flash‑Next replaces dense attention with sparse alternatives and stabilizes training via Gated Residual connections.

QSA delivers large inference speedups: at a 1 M context length it is 7.6× faster than dense attention in prefill and 4.9× faster in decode at the kernel level.

Measured on the kernel‑level benchmark described in §2.1.1.

Loss and downstream accuracy do not always move together: enlarging the n‑gram vocabulary monotonically lowers loss while accuracy quickly saturates, so under a fixed parameter budget the loss optimum diverges from the accuracy optimum (see Tables 8 & 9).

Predicting the residual read and write weights from the residual state yields only a marginal loss reduction but a clear benchmark gain (§2.2).

Restricting each block to the two highest‑gated residual branches is almost free in pre‑training loss yet degrades performance with further training (§2.2).

Removing positional encoding from the full‑attention layers is indistinguishable during pre‑training but harms generation quality at later stages (§2.1.1).

Muon introduces engineering costs: per‑parameter FLOPs depend on matrix shape rather than count, the gradient buffer is repartitioned by estimated orthogonalization cost, and the step fragments into many small kernels, which we capture in a CUDA graph (§3.1). We set the Newton–Schulz iteration to eight steps for added stability.

During inference, prefill is dominated by attention over the whole context, which QSA accelerates by compressing the key sequence; decode is dominated by memory traffic, so GDN layers keep a fixed‑size recurrent state, GR drops the branch‑mixing operator $H_{\text{res}}$, and the residual state stores values in FP8.

Optimization applies Muon only to two‑dimensional weight matrices; other components (input/n‑gram embeddings, output head, MoE router, low‑rank GR projections) remain on AdamW because orthogonalization is impractical there. Fused parameters are split before orthogonalization to avoid mixing singular directions across unrelated sub‑blocks (§3.1).

We refit the scaling law (Kaplan et al., 2020) for the Qwen 3.5 series (§ 3.2), which predicts a larger batch size and learning rate. Both predictions are verified: the larger batch improves parallel throughput, and the larger learning rate speeds convergence. Ramping the batch size over early training adds 18.8% more optimizer steps without benefit, so we keep the target batch size throughout.

Training‑stability stress tests raise the learning rate fourfold. The previous architecture spikes frequently, whereas the new recipe remains stable throughout (§ 3.3). Isolating the gate in GR on a single‑variable pair confirms it as the key contributor to the stability margin over the Qwen 3.5 architecture.

QSA delivers up to 7.6× inference speedup while GR ensures training stability, and the refitted scaling law unlocks higher throughput at larger batch sizes.

Architecture Comparison

Performance and efficiency comparison of attention architectures across benchmarks.

**Table 1.** Performance comparison of different attention architectures across various benchmarks.

For efficiency, the authors replace the dense GDN kernel with FlashQLA, a TileLang‑based fused linear‑attention implementation. On NVIDIA GPUs FlashQLA delivers a 2–3× forward speedup and roughly a 2× backward speedup compared to the prior FLA Triton kernel.

Residual Connection Overview

Explain why widening residuals and adding learned mixing improves training stability.

Residual connections give every block a direct path to the network output, but pre‑normalization attenuates the signal each layer receives, forcing early features to compete with later ones.

Instead of a single residual vector, we keep $nr$ parallel branches and read a weighted sum of them, then write the block output back to one branch in a round‑robin fashion.

How does this simplified AltUp differ from a plain residual connection?

A plain residual adds a single vector to the input. AltUp keeps $nr$ parallel branches, learns a weighted sum of them for the input, and writes the output back to a rotating branch, thereby widening the information flow without extra matrix multiplications.

Compute the weighted sum: $x^{(0)} = 0.7\cdot[1,2,3,4] + 0.3\cdot[5,6,7,8] = [2.2,\,3.2,\,4.2,\,5.2]$.

Since $\ell=0$, the write‑back branch is $(0\bmod2)+1 = 1$, i.e., the first branch.

Update $R^{(1)}$: replace the first branch with $y^{(0)}=x^{(0)}$, keep the second unchanged. $R^{(1)}=\begin{bmatrix}2.2&3.2&4.2&5.2\\5&6&7&8\end{bmatrix}$.

Next layer $\ell=1$ would read $x^{(1)} = 0.7\cdot[2.2,3.2,4.2,5.2] + 0.3\cdot[5,6,7,8] = [3.04,\,4.04,\,5.04,\,6.04]$.

Write‑back branch for $\ell=1$ is $(1\bmod2)+1 = 2$, so the second branch becomes $[3.04,4.04,5.04,6.04]$.

The alternating write‑back spreads information across branches, allowing later layers to see a mixture of earlier outputs while preserving the cheap per‑layer cost.

We keep $nr$ residual branches but now learn three operators that mix the branches for reading, combine the block output back into the branches, and exchange information among branches.

Why is Hyper‑Connections more expressive than the simple weighted‑sum read used in the simplified AltUp?

AltUp only learns static scalar weights $h_i$ for reading and writes to a single branch. HC learns three separate operators: a mixing vector $H_{\text{mix}}$, a mixing matrix $H_{\text{res}}$, and a combine vector $H_{\text{combine}}$, each of which can adapt based on the current residual state via the data‑dependent terms in Eq. 26. This allows the network to dynamically re‑weight, exchange, and inject information across branches, not just a fixed linear combination.

Normalize $R^{(0)}$ (already unit‑norm), so $\phi(R)=R$.

Compute $H_{\text{mix}} = [0.5,0.5] + 0.01\odot R\,I = [0.51,0.51]$ (element‑wise addition).

Read input: $x^{(0)} = H_{\text{mix}}\,R^{(0)} = 0.51\cdot[1,0,0,0] + 0.51\cdot[0,1,0,0] = [0.51,0.51,0,0]$.

Assume $F^{(0)}$ is identity and $\text{Norm}$ does nothing, so $y^{(0)} = x^{(0)} = [0.51,0.51,0,0]$.

Compute $H_{\text{combine}} = [1,0] + 0.01\odot R\,I = [1.01,0.01]$.

Update residual: $R^{(1)} = I\,R^{(0)} + H_{\text{combine}}\,y^{(0)} = R^{(0)} + [1.01,0.01]^\top \cdot [0.51,0.51,0,0]$ resulting in $R^{(1)}=\begin{bmatrix}1+0.5151 & 0+0.5151 & 0 & 0\\0+0.0051 & 1+0.0051 & 0 & 0\end{bmatrix}$.

Even with tiny data‑dependent terms, HC can subtly shift the residual branches, illustrating how the learned operators can gradually steer information flow beyond the static baseline.

**Figure 7.** Cross-layer paths added by GR. Each row corresponds to one residual branch; a connection runs from the sublayer that wrote into that branch to a later sublayer that reads it back. Vertical position encodes the number of layers skipped; line width and opacity encode $\Delta_{uv}$ (Eq. (37)), the additional share of the reader's input supplied by this writer compared to a single residual stream. Shaded regions denote softmax-attention layers, every fourth in this hybrid; the rest are GDN, and sublayers are named accordingly (L00.GDN, L03.attn, L00.mlp). Readers are named where they are softmax-attention sublayers, which is where the long-range paths land. One branch carries long-range paths (all connections on $b_0$ originate at layer 0 and land past layer 10), while the other three stay local (median skips of 1.2–3.5 layers). The extra width is spent on a small number of specific paths, mostly preserving early GDN output across depth and delivering it to the softmax-attention layers. Thresholds and counts are given in the text.

Residual Design Ablation

We test each residual tweak by removing it and measuring loss and benchmark scores.

To assess the contribution of each residual design choice we start from the static operators and progressively add expressiveness only where it yields measurable benefit.

**Table 5.** Comparison of different residual connection methods across various benchmarks.

Static residual connections (the $\\lambda^{\\star}=0$ case) lower training loss by 0.021 relative to the pre‑norm baseline.

Table 5 shows a loss reduction of 0.021 when moving from Pre‑norm to the static variant.

Making the residual operators data‑dependent (dynamic) further reduces loss by 0.002 compared with the static version.

Table 5 records a loss drop of 0.002 when switching from static to dynamic residuals.

Static operators increase the average benchmark score by 1.58 points over the pre‑norm baseline.

The average column rises from the baseline to 56.61 for the static variant, a gain of 1.58.

Introducing data‑dependence in both read ($H_{\\text{mix}}$) and write ($H_{\\text{combine}}$) adds a further 1.98‑point boost in average score.

Table 5 shows the dynamic variant achieving an average of 58.59, 1.98 points higher than the static case.

Refining the read operator $H_{\\text{mix}}$ from a single scalar per branch to a per‑branch‑per‑channel weight yields a noticeable gain, whereas applying the same refinement to the write operator $H_{\\text{combine}}$ brings almost no improvement.

Predicting the residual operators from all $n_{r}=4$ branches outperforms using only the last branch or pooling the branches first; normalizing each branch separately with a group RMSNorm adds another boost.

Adding the $n_{r}\\times n_{r}$ mixing operator $H_{\\text{res}}$ provides negligible benefit once the read and write paths are already expressive.

Inference Efficiency of Gated Residuals

GR matches Full AttnRes loss while halving inference memory traffic.

GR matches the loss of Full AttnRes while cutting inference memory traffic.

Table 6 shows GR loss 1.762 equal to Full AttnRes 1.762; Figure 10 confirms comparable training loss across learning rates.

Table 6 compares residual designs at 28 layers, showing that GR attains the lowest loss (1.762), matching Full AttnRes, while block‑summarizing variants incur small overheads (0.008–0.011).

**Figure 10.** **Training loss under stress.** The 28-layer 25B-A3B MoE at a constant learning rate: the Qwen3.5 structure under AdamW, the same structure under Muon, and Muon with GR. Bold lines are a moving average over the faint per-step trace. GR enables more stable training.

**Figure 11.** Gradient norm and activations under stress. The same three runs as Fig. 10 (a). The activation in (b) is averaged over layers. Gradient Residual reduces both the frequency and magnitude of gradient-norm spikes, as well as the magnitude of activation outliers (Fig. 11).

We attempted to sparsify the GR read by selecting only the two branches with highest gate values; pre‑training loss and benchmarks were unchanged, but post‑training quality degraded, so the approach was abandoned.

N-gram Embedding

N‑gram embedding adds context‑aware memory with negligible compute cost.

Scaling model capacity by adding a separate memory dimension is attractive, but naive token‑wise embeddings double the per‑token FLOPs. The N‑gram embedding trick sidesteps this by conditioning lookups on short local contexts, keeping the extra compute essentially zero.

Instead of a single token ID fetching a vector, each token queries a tiny table of its preceding n‑gram (e.g., bigram, trigram) and adds the retrieved vector to its representation.

How does N‑gram embedding differ from the standard token embedding used in transformers?

Standard token embedding maps a single token ID to a vector, ignoring surrounding tokens. N‑gram embedding maps a short sequence (e.g., the previous token plus the current one) to a vector, so the retrieved memory already encodes local context before any attention is applied.

At position 2 we form the bigram (t₁,t₂) and retrieve embedding e₀ from slot 0.

At position 3 we form (t₂,t₃) and retrieve embedding e₁ from slot 1.

At position 4 we form (t₃,t₄) and retrieve embedding e₂ from slot 2.

Each token’s hidden vector $h_i$ is updated to $h_i$ ← $h_i$ + $g_i$ ⊙ $e_{slot}$, where $g_i$ is the gating scalar from the GR module.

The lookup cost is constant per token regardless of vocabulary size, because the hash reduces the combinatorial bigram space to a fixed‑size table.

A single N‑gram embedding layer placed at the second transformer block yields the lowest loss and the strongest downstream scores; adding more layers gives no consistent benefit.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers