xHC: Expanded Hyper-Connections

Xiangdong Zhang, Xiaohan Qin, Sunan Zou, Tuo Dai, Xiaoming Shi, Huaijin Wu, Yebin Yang, Zhuo Xia, Shaofeng Zhang, Lin Yao, Yuliang Liu, Yu Cheng, Junchi Yan

xHC enables efficient, large-scale residual-stream expansion by combining temporal feature augmentation with sparse residual updates.

How can we scale Transformer residual streams to many parallel channels without hitting the memory-bandwidth bottlenecks that plague existing dense expansion methods?

Transformer models typically use a single residual stream, limiting cross-layer information flow. While expanding to multiple streams (Hyper-Connections) improves capacity, scaling beyond four streams hits diminishing returns due to redundant write-back signals and cubic computational costs. xHC (Expanded Hyper-Connections) solves this by enriching write-back signals with local temporal context and using a sparse architecture that updates only a subset of streams. This decouples the benefit of larger memory from the cost of residual mixing. On 18B and 28B MoE models, xHC consistently outperforms state-of-the-art baselines, achieving lower training loss while requiring significantly less compute to reach equivalent performance.

Paper Primer

The core bottleneck in existing multi-stream models is a "benefit-cost imbalance": each layer injects only a single signal into the residual state, making additional streams redundant, while the cost of mixing these streams scales cubically with the number of streams $N$. xHC addresses this by introducing temporal feature augmentation to diversify the write-back signal and a sparse residual-stream architecture that limits mixing to $k=4$ active streams, effectively reducing the cost from $O(N^3C)$ to $O(k^3C)$.

xHC makes residual-stream expansion a practical scaling axis, significantly improving compute efficiency over existing methods.

Scaling-law experiments show that to reach the same loss, vanilla Transformers require 1.50× and mHC requires 1.19× the compute of xHC. xHC improves average downstream scores on an 18B MoE model by 4.0 points over mHC while adding only ~4% training FLOPs.

xHC-Flash maintains the performance gains of large-N expansion while reducing memory traffic to levels comparable to small-N baselines.

xHC-Flash-4sub reduces per-sublayer memory traffic from 73.5C to 40C, nearly matching the 34C required by mHC at N=4. Retains nearly all performance gains of full xHC despite the significant reduction in I/O overhead.

Why does xHC use a sparse residual update instead of just updating all streams?

Updating all $N$ streams leads to a cubic computational cost $O(N^3C)$ for residual mixing. By updating only $k=4$ streams while keeping the read path dense, xHC maintains cross-layer information flow without the prohibitive cost of full-state mixing.

What is the role of temporal feature augmentation in this architecture?

It enriches the write-back signal by incorporating local contextual information from neighboring tokens via causal depthwise convolutions. This prevents the information bottleneck where additional streams would otherwise receive redundant, single-component signals.

Introduction and Motivation

The section exposes why dense multi‑stream residual models stall beyond four streams.

Existing multi‑stream residual architectures (mHC) quickly hit a performance‑efficiency wall: they process every stream densely, which inflates memory traffic and compute without delivering proportional gains.

Instead of a single residual vector flowing through the network, the model maintains N parallel streams that can store distinct historical information.

mHC stabilizes dense multi‑stream residuals by learning a manifold‑constrained mixing matrix that softly routes information among the N streams.

The first bottleneck is informational: as N grows, each layer still contributes only one write‑back signal, so additional streams receive increasingly redundant data.

The second bottleneck is computational: generating the dense residual‑mixing matrix $H_{\text{res}}$ requires $O(N^3 C)$ operations, making large‑N expansion prohibitively expensive.

Dense multi‑stream models hit a memory‑bandwidth ceiling that prevents practical scaling beyond N = 4.

The xHC Architecture

We introduce xHC, a sparse routing and temporal augmentation scheme that makes large‑N residual expansion efficient.

Scaling the number of residual streams (N) in existing multi‑stream designs quickly hits diminishing returns: extra streams add little new information while the residual‑mixing cost grows cubically.

xHC augments the write‑back signal and sparsifies residual mixing so that many streams can be added without exploding compute.

How does xHC differ from a naïve dense expansion of HC?

Naïve HC would compute a full N×N residual matrix at every layer, incurring O(N³C) cost and using only a single write‑back component. xHC instead (1) enriches the write‑back with orthogonal local features, and (2) limits residual mixing to k active streams, reducing the cost to O(k³C) while still accessing all N streams during the read phase.

Router scores the four streams as s = [0.9, 0.2, 0.7, 0.1] and selects the fixed stream 1 plus the top‑K stream 3, so I = (1,3).

Dense read aggregates all streams: input = $\Sigma_{i}$ $H^{pre}_i$ · $x_i$, producing a 2‑dim vector.

Temporal Feature Augmentation generates three depthwise convolutions (kernel sizes 2,3,4) yielding three additional 2‑dim components.

Gram–Schmidt orthogonalizes the four components, producing orthogonal basis vectors v₁…v₄.

Post‑mapping $H^{post}$ (size 2×4) combines the four basis vectors into two write‑back updates $\Delta$x₁ and $\Delta$x₃.

Residual mixing $H^{res}$ (2×2) mixes the active streams, and the updates are scattered back into the full 4‑stream state.

Even with only two active streams, the orthogonalized augmentations give each stream a distinct signal, preventing redundancy as N grows.

Local causal convolutions inject neighboring‑token information into the write‑back, turning a single output into a richer set of components.

Why not simply compute a separate output for each stream instead of augmenting with convolutions?

Computing an independent output per stream would multiply the layer FLOPs by N, defeating the efficiency goal. Convolution‑based augmentation reuses the same output while cheaply adding diverse signals.

The architecture updates only k streams per layer, but every layer can still read from all N streams, achieving a cheap yet expressive residual network.

How does the sparse update avoid breaking information flow across layers?

Because the read phase remains dense: every layer still aggregates information from all N streams via the pre‑mapping. Only the write‑back is sparse, so the network retains long‑range cross‑stream pathways while cutting the expensive mixing cost.

**Figure 3.** Overview of xHC. (a) A standard Transformer layer maintains a single residual stream. (b) mHC expands the residual state into $N=4$ streams with dense residual mixing and write-back. (c) xHC scales to $N=16$ with only $k=4$ active streams: it reads all streams, applies the sublayer $\mathcal{F}$ (Attn/MLP), augments MLP outputs, and sparsely writes back to selected streams. Blue/orange streams denote fixed/routed active streams. Restricting residual mapping to the $k$ active streams reduces the dominant residual-mapping generation cost from $O(N^3C)$ to $O(k^3C)$.

Forward pass of one xHC sublayer

Together, temporal augmentation supplies richer write‑back signals while sparse residual mixing keeps the computational budget low, enabling practical scaling to dozens of streams.

Parameterization and Implementation

Method details define the gated‑projection generators that enable sparse, coordinated stream updates.

Dense multi‑stream processing in mHC quickly exhausts compute and memory, so the model stalls. The xHC trick is to replace that dense flow with lightweight, gated‑projection generators that activate only the needed streams.

Each stream’s update is produced by a tiny linear projection whose contribution is modulated by a learnable gate $\alpha$, so the mapping starts almost static and gradually becomes data‑dependent.

Flatten $X^{\ell}$ to a 4‑element vector, apply RMSNorm → $\mathbf{z}=(0.5,\,0.5,\,0.5,\,0.5)$.

Pre‑mapping: compute $\alpha_{\text{pre}}\mathbf{z}W_{\text{pre}}+b_{\text{pre}} \approx 0.01\cdot\mathbf{z}W_{\text{pre}}$, then sigmoid → $H_{\text{pre}}^{\ell}\approx(0.502,\,0.502)$ (values stay near 0.5).

Reshape $\tilde{x}_{\text{active}}$ (same as $\mathbf{z}$) to $2\times2$, multiply by $W_{\text{res}}$, add bias, exponentiate, apply SK → $H_{\text{res}}^{\ell}\approx(0.25,\,0.25,\,0.25,\,0.25)$.

Post‑mapping: similar projection with $W_{\text{post}}$, sigmoid, then multiply by 2 → $H_{\text{post}}^{\ell}\approx(0.02,\,0.02)$.

Combine the three generators (e.g., additive) to obtain the final stream update; the tiny gates keep the update close to the bias‑only baseline.

The example shows that with $\alpha=0.01$ the generators hardly move away from their bias terms, guaranteeing a stable start; as training progresses $\alpha$ grows and the projections become fully expressive.

How does this scalar gating differ from the per‑channel FiLM modulation used in other gated networks?

FiLM multiplies each channel by its own learned scale, requiring $C$ extra parameters per gate. Here a single scalar $\alpha$ modulates the *entire* flattened stream, adding only one parameter per generator. This extreme sparsity is what lets xHC scale to many streams without blowing up memory.

Normalize the full state $X^{\ell}$ with RMSNorm and flatten.

Compute $H_{\text{pre}}^{\ell}$ using the gated‑projection pre‑mapper.

Extract the active subset, normalize to $\tilde{x}_{\text{active}}$, then compute $H_{\text{res}}^{\ell}$ and $H_{\text{post}}^{\ell}$.

Apply the temporal convolution branches: for each kernel $w_{j}$ (sizes 4, 8, 12) compute $g_{j}[t]=\sum_{i=0}^{\kappa_{j}-1} w_{j}[i]\odot\text{out}[t-i]$.

Combine the pre, residual, post, and convolution outputs to form the updated stream representation.

Pass the result to the next sublayer or downstream head.

Empirical Results

xHC consistently outperforms mHC and vanilla baselines across compute scales.

xHC improves average downstream score by 4.0 points over mHC on 18B MoE models.

Table 1 shows average scores 48.8 for xHC versus 44.8 for mHC.

**Figure 1.** Expansion efficiency: loss vs. FLOPs on a 2.5B MoE model (details in Table 6).

**Figure 2.** xHC delivers broad gains at 18B scale. (a) xHC achieves lowest training loss. (b) The loss improvement translates into a significantly higher average downstream score across benchmarks in Table 1. (c) xHC improves representative benchmarks across reasoning, knowledge, and code.

**Figure 4.** Scaling-law comparison. xHC traces a consistently lower loss curve than both mHC and the vanilla residual baseline across training compute.

Ablations and Optimizer Scaling

Systematic removals reveal which xHC components drive loss reduction and efficiency.

This section isolates each xHC design decision by removing it from a 10B MoE baseline and measuring the impact on validation loss and compute cost.

Temporal feature augmentation lowers validation loss by 0.014.

Loss drops from 1.998 (dense mHC) to 1.984 when the augmentation is added at $N=16$.

Sparse residual‑stream updates slash FLOPs overhead by 16.8 %.

Overhead falls from 20.1 % to 3.3 % while loss stays at 1.983.

**Figure 5.** Information bottleneck ablation. The gain grows with $N$.

At $N=16$, the loss gap reaches –0.012, the deepest improvement observed.

Figure 5 shows the red “+ Temp Aug” line descending to roughly –0.012 at $N=16$.

Removing Dense Read raises loss by 0.002.

Loss climbs from 1.983 (full xHC) to 1.985 when Dense Read is omitted.

Removing Fixed streams raises loss by 0.003.

Loss climbs from 1.983 to 1.986 when Fixed streams are omitted.

Active‑stream budget $k=2$ inflates loss by 0.008.

Validation loss is 1.991 versus the $k=4$ baseline of 1.983.

Active‑stream budget $k=8$ trims loss by 0.001.

Validation loss improves to 1.982, a slight gain over the $k=4$ baseline.

Sigmoid routing improves loss by 0.005 compared with Softmax.

Loss with Sigmoid is 1.983 versus 1.988 with Softmax.

**Table 2.** Ablation study. Parentheses show extra training FLOPs over vanilla baseline. “Sparse” is the sparse architecture, “D. Read” is Dense Read, and “Fixed” is the number of always-active streams.

Finally, we verify that xHC remains effective when paired with the Muon optimizer: Gram–Schmidt orthogonalization is unnecessary, and Muon is applied only to backbone matrices, leaving xHC‑specific projections under AdamW.

Memory and Throughput Analysis

We revisit the core premise: dense multi‑stream residuals waste memory traffic, while xHC uses sparse coordinated reads to keep traffic low.

Maintaining many residual streams inflates memory traffic, which often dominates runtime cost. xHC mitigates this by activating only a few streams per sublayer, but the remaining full‑state reads still pose a bottleneck.

A memory‑bound bottleneck occurs when the time to move data dominates over the time to compute it, so adding arithmetic work has little impact on overall speed.

**Table 4.** Per-token memory access of residual-stream operations (sublayer function I/O excluded) Totals averaged per sublayer. *Once per block. †Includes MLP-side conv (+C R, +3C W) and Flash input correction (+2C R, +C W). ‡Amortized over 4 sublayers.

The provided image does not contain a table label or caption.

xHC and xHC‑Flash achieve the lowest validation loss of 1.983.

Table 5 shows 1.983 for both methods, compared with 2.029 for mHC (N = 4).

Implementation Details

Implementation details of xHC‑Flash focus on kernel fusion and memory‑traffic reduction.

This section details the concrete engineering that turns the xHC idea into the xHC‑Flash implementation used in large‑scale models.

**Require:** $X \in \mathbb{R}^{N \times C}$ 1: 2: /* Joint pre-forward (once per block) */ 3: $s, \mathcal{H}_{\text{A}}^{\text{pre}}, \mathcal{H}_{\text{M}}^{\text{pre}} \leftarrow \text{JointFullStateMap}(\text{flatten}(X))$ 4: $\mathcal{I}, p \leftarrow \text{Route}(s; m, k); \quad X_{\text{act}} \leftarrow \text{Gather}(X, \mathcal{I})$ // $k$ active streams 5: $\text{inp}_{\text{A}}, \text{inp}_{\text{M}} \leftarrow \text{JointDenseRead}(X, \mathcal{H}_{\text{A}}^{\text{pre}}, \mathcal{H}_{\text{M}}^{\text{pre}})$ 6: 7: /* Attention (no residual mixing) */ 8: $\mathcal{H}_{\text{A}}^{\text{post}} \leftarrow f_{\text{post}}^{\text{A}}(X_{\text{act}})$ // $\mathbb{R}^{k \times 1}$ 9: $\text{out}_{\text{A}} \leftarrow \text{Attn}(\text{inp}_{\text{A}})$ 10: $X_{\text{act}} \leftarrow X_{\text{act}} + p \odot (\mathcal{H}_{\text{A}}^{\text{post}} \text{out}_{\text{A}})$ 11: $\alpha \leftarrow \sum_{j=1}^{k} \mathcal{H}_{\text{M}, \mathcal{I}_j}^{\text{pre}} p_j \mathcal{H}_{\text{A}, j}^{\text{post}}$ 12: $\text{input}_{\text{M}} \leftarrow \text{inp}_{\text{M}} + \alpha \text{out}_{\text{A}}$ // exact dense-read correction 13: 14: /* MLP (with residual mixing + temporal aug) */ 15: $\mathcal{H}_{\text{res}}, \mathcal{H}_{\text{M}}^{\text{post}} \leftarrow f_{\text{res}}(X_{\text{act}}), f_{\text{post}}^{\text{M}}(X_{\text{act}})$ // $\mathbb{R}^{k \times k}, \mathbb{R}^{k \times K_r}$ 16: $\text{out}_{\text{M}} \leftarrow \text{MLP}(\text{input}_{\text{M}})$ 17: $X_{\text{act}} \leftarrow \mathcal{H}_{\text{res}} X_{\text{act}} + p \odot (\mathcal{H}_{\text{M}}^{\text{post}} \text{TempAug}(\text{out}_{\text{M}}))$ 18: 19: $X \leftarrow \text{Scatter}(X, \mathcal{I}, X_{\text{act}})$ 20: **return** $X$

High‑level pseudocode for one xHC‑Flash block (forward pass only).

The first stage computes all routing and projection coefficients while reusing a single set of statistics, so the model never materializes full‑state normalized tensors.

The second stage applies the previously generated mappings in a handful of fused kernels, eliminating redundant reads and writes of $kC$‑dimensional tensors.

Table 5 reports validation loss and estimated I/O per sublayer for the vanilla baseline, mHC ( $N=4$ ), xHC ($N=16$, $k=4$), and two xHC‑Flash variants.

Across the two sublayers of a Transformer block, the router, pre‑mappings, dense reads, and post‑mappings are packed into two full‑state passes, each followed by a small correction kernel.

End‑to‑end wall‑clock training on the 18 B MoE model shows xHC‑Flash adds ~6 % overhead versus the vanilla baseline; xHC‑Flash‑4sub adds ~11 % on top of mHC. Inference pre‑fill at 2 K tokens incurs only a 1.3 % latency increase over mHC.

Additional Ablations

Ablation results quantify the impact of temporal augmentations and convolution branches on validation loss.

We evaluate three ablations on temporal feature augmentation and two on the number of temporal convolution branches, all under the $10B$ MoE setting.

xHC default achieves the lowest validation loss among temporal‑augmentation variants.

Table 11 reports a loss of 1.983 for the default configuration.

Removing Gram–Schmidt orthogonalization raises the loss to 1.984; the increase is negligible at $10B$ scale but at $18B$ the lack of orthogonalization leads to training instability, so the step is retained for large‑scale robustness.

Adding three multi‑scale temporal branches improves validation loss by 0.005 over the single‑scale variant.

Loss drops from 1.989 to 1.984 (Table 12).

Temporal augmentations after the MLP layer and multi‑scale convolution branches provide consistent, modest validation‑loss reductions without extra overhead.

xHC-Flash Optimization

xHC‑Flash recovers exact sublayer inputs with only a scalar‑vector correction, avoiding dense reads.

Recall that multi‑stream residual models (mHC) waste compute by reading every stream after each sublayer; xHC‑Flash eliminates that waste by reusing a pre‑computed base readout and applying only a sparse correction.

xHC‑Flash removes the attention‑side residual mixing so that later sublayer inputs can be reconstructed exactly from a single base readout plus a cheap correction for the active streams.

Compute the dense base readout $\text{inpM}= \sum_{i=1}^{4} H^{\text{pre,MLP}}_i\, x^{(0)}_i$ (one scalar‑vector multiply per stream, done once).

Form $\alpha = 0.2\!\times\!1\!\times\!0.1 \;+\; 0.2\!\times\!1\!\times\!0.3 = 0.08$.

Apply the correction $\alpha\,\text{outAttn}=0.08\,\text{outAttn}$ and add to $\text{inpM}$ to obtain the MLP input for the second sublayer.

The entire correction collapses to a single scalar–vector multiply, so the second sublayer never reads the full $4$‑stream state again.

How does xHC‑Flash differ from the original xHC dense‑read approach?

Original xHC recomputes a full dense read after every sublayer, costing $O(NC)$. xHC‑Flash computes the dense read once, then for each subsequent sublayer adds only a scalar‑vector (two‑sublayer) or an active‑stream sum (four‑sublayer) that depends on the $k$ changed streams, reducing the per‑sublayer cost to $O(kC)$.

The four‑sublayer variant generalises the scalar correction by keeping a separate pre‑mapping for each sublayer while sharing the same routing decision across the group.

Compute $\text{base}^{(2)}_{\neg I}=0.5\,(x^{(0)}_{2}+x^{(0)}_{4})$ (unchanged streams).

Form $\text{inp}^{(2)}_{\text{act}}=0.5\,(x^{(1)}_{1}+x^{(1)}_{3})$, where each $x^{(1)}_{I_j}$ already contains its first write‑back.

Combine: $\text{input}^{(2)}=\text{base}^{(2)}_{\neg I}+\text{inp}^{(2)}_{\text{act}}$, yielding the exact same vector that a full dense read would have produced after two sublayers.

Because each active stream carries its accumulated updates, later sublayers never need to materialise a separate delta buffer; the correction stays a simple sum of current active states.

The correction is exact as long as (1) routing and pre‑mappings are fixed from the entry state, (2) residual mixing is deferred until the final MLP, and (3) only the selected active streams are updated. Under these conditions xHC‑Flash reproduces the full‑xHC sublayer inputs without any $O(NC)$ dense reads, while still sharing routing decisions across a short window.

References

Supplementary references, model tables, and detailed training settings.

This appendix collects the full bibliography, the two configuration tables, and all low‑level hyperparameter choices omitted from the main text.

Table 6 enumerates the parameter counts, layer counts, and optimizer settings for the 2.5 B, 10 B, and 28 B model scales, ensuring that each comparison in the paper isolates the effect of the residual‑stream design.

The xHC configuration fixes the expansion rate at $N\!=\!16$, with $k\!=\!4$ active streams per layer, of which $m\!=\!2$ are permanently active and the remaining $k-m\!=\!2$ are selected by a sigmoid‑activated linear router applied to the LayerNorm‑normalized residual state.

Temporal feature augmentation adds $r\!=\!3$ causal depthwise‑convolution branches with kernel sizes $\{4,8,12\}$ after each MoE FFN, followed by Gram–Schmidt orthogonalization; Sinkhorn normalization runs for 20 iterations and rows exceeding a sum of 1 are clamped.

The N‑sweep experiments use the 2.5 B configuration from Table 6 while varying $N$, $k$, and $m$ as shown in Table 7; for the dense baseline (mHC) all $N$ streams are always active.

**Table 7.** $N$-sweep configuration for xHC.

Before the final RMSNorm and unembedding layer, the multi‑stream hidden state is reduced to a single vector by summing the $N$ residual streams for each token, providing a compact representation for the language‑model head.

Section 4.6 employs the Muon optimizer on all 2‑D weight matrices (attention, MLP/MoE, router projections) while AdamW handles embeddings, normalizations, and xHC‑specific parameters; Muon uses momentum $\beta\!=\!0.95$, 5 Newton‑Schulz iterations, and targets an AdamW‑RMS equilibrium of 0.2.

Evaluation follows the OpenCompass framework, using perplexity‑based selection for multiple‑choice benchmarks (MMLU, CMMLU, etc.) and generation‑based scoring for free‑form tasks (HumanEval, LCBench, etc.).

Scaling Law Configurations

Details of the scaling‑law experiment setup and fitted parameters.

We train four model sizes for each method—vanilla baseline, mHC, and xHC—using identical recipes and a compute budget ranging from $1.7 \times 10^{19}$ to $4.0 \times 10^{20}$ FLOPs. All configurations follow the DeepSeekMoE‑style template (Table 6) with $144$ routed experts, top‑8 routing, one shared expert, and a leading dense layer, while varying width, depth, and expert FFN size. Scaling laws are fit to the final language‑modeling loss using a shifted power‑law $L(C)=A\,C^{-\alpha}+E$ with $E=0.72$.

**Table 6.** Model configurations and training hyperparameters for the main model scales. The 2.5B setting is used for $N$-sweep experiments, the 10B setting for ablations, and the 18B/28B settings for main results.

**Table 8.** Scaling-law model configurations. All models use context length 8192, RoPE [39] ($\theta=50000$), 144 routed experts, top-8 routing, 500 warmup steps, and the same data mixture.

**Table 9.** Fitted scaling-law parameters.

Parameter and FLOP Analysis

Training FLOPs and parameter overhead of xHC versus mHC are quantified.

This appendix derives the extra parameters and FLOPs introduced by the residual‑stream modules (xHC and mHC) and reports their impact on training cost for the 18B and 28B model families.

xHC with $N\!=\!16$, $k\!=\!4$ uses 7.3× fewer parameters per layer than dense mHC at the same expansion rate.

Per‑layer counts: $P_{xHC}=1256\,C$ vs. $P_{mHC}=9216\,C$ (Section C.1).

The table compares parameter and FLOP overheads for different model configurations (18B and 28B) using mHC and xHC architectures. It lists values for $P$/layer, $\Delta P\%$, and $\Delta F\%$ across rows representing mHC ($N=4$), mHC ($N=16$), and xHC ($N=16, k=4$).

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers