DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
DeepSeek-AI, Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chengyu Hou, Chenhao Xu, Chenze Shao, Chong Ruan, Conner Sun, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Donghao Li, Dongjie Ji, Erhang Li, Fang Wei, Fangyun Lin, Fangzhou Yuan, Feiyu Xia, Fucong Dai, Guangbo Hao, Guanting Chen, Guoai Cao, Guolai Meng, Guowei Li, Han Yu, Han Zhang, Hanwei Xu, Hao Li, Haofen Liang, Haoling Zhang, Haoming Luo, Haoran Wei, Haotian Yuan, Haowei Zhang, Haowen Luo, Haoyu Chen, Haozhe Ji, Hengqing Zhang, Honghui Ding, Hongxuan Tang, Huanqi Cao, Huazuo Gao, Hui Qu, Hui Zeng, J Yang, JQ Zhu, Jia Luo, Jia Song
DeepSeek-V4 introduces hybrid attention and manifold-constrained connections to enable efficient 1M-token context.
How does the DeepSeek-V4 series achieve million-token context intelligence while maintaining efficiency through architectural and infrastructure optimizations?
The quadratic computational cost of standard attention mechanisms creates a bottleneck that prevents Large Language Models (LLMs) from scaling to ultra-long contexts and complex reasoning tasks. DeepSeek-V4 addresses this by combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to drastically reduce Key-Value (KV) cache size and inference FLOPs, while using Manifold-Constrained Hyper-Connections (mHC) to stabilize deep model training. In a 1-million-token context, DeepSeek-V4-Pro achieves 27% of the single-token inference FLOPs and 10% of the KV cache size compared to the previous DeepSeek-V3.2 architecture.
Paper Primer
The core architectural innovation is a hybrid attention mechanism that interleaves two compression strategies: CSA, which compresses KV entries and applies sparse attention, and HCA, which uses more aggressive compression without sparse selection. This is paired with mHC, which constrains residual transformations to a manifold of doubly stochastic matrices to ensure non-expansive signal propagation across deep layers.
DeepSeek-V4-Flash achieves extreme long-context efficiency.
In a 1M-token context, the model requires only 10% of the single-token FLOPs and 7% of the KV cache size relative to DeepSeek-V3.2. ~90% reduction in compute and ~93% reduction in memory footprint for long-context inference.
To support this architecture, the authors implemented a fine-grained expert parallelism scheme that fuses communication and computation into a single pipelined kernel, effectively hiding communication latency beneath computation. They also adopted the Muon optimizer, utilizing hybrid Newton-Schulz iterations to achieve faster convergence and improved stability compared to standard AdamW.
Why is the hybrid attention approach necessary instead of just using standard sparse attention?
Standard sparse attention alone does not sufficiently address the memory bottleneck of ultra-long sequences. By interleaving CSA and HCA, the model balances the need for local dependency modeling (via sliding-window attention) with the extreme compression required to keep 1M-token contexts computationally feasible.
What is the primary role of the Manifold-Constrained Hyper-Connections (mHC)?
mHC strengthens residual connections by constraining the residual mapping matrix to the Birkhoff polytope, which bounds the spectral norm to 1. This prevents signal explosion and ensures numerical stability when stacking deep layers, a common failure point in standard hyper-connection implementations.
DeepSeek-V4 demonstrates that architectural co-design—specifically fusing communication-computation kernels with aggressive KV cache compression—can make million-token context windows a standard, efficient capability rather than a prohibitive research luxury.
Executive Summary
A preview of DeepSeek‑V4 series delivering million‑token context with strong efficiency gains.
DeepSeek‑V4 introduces a hybrid attention stack that merges Compressed Sparse Attention and Heavily Compressed Attention, augments residual pathways with Manifold‑Constrained Hyper‑Connections, and trains with the Muon optimizer for faster convergence. The series is pre‑trained on more than 32 T high‑quality tokens and refined through a dedicated post‑training pipeline, culminating in a maximum‑effort mode (DeepSeek‑V4‑Pro‑Max) that pushes state‑of‑the‑art performance.
The series comprises large‑scale Mixture‑of‑Experts language models that scale to a one‑million‑token context by compressing the key‑value cache and parallelizing expert computation.
**Figure 1 |** Left: benchmark performance of DeepSeek-V4-Pro-Max and its counterparts. Right: inference FLOPs and KV cache size of DeepSeek-V4 series and DeepSeek-V3.2.
In a one‑million‑token setting DeepSeek‑V4‑Pro consumes only 27 % of the single‑token inference FLOPs and 10 % of the KV cache compared with DeepSeek‑V3.2, delivering substantial performance and efficiency gains.
The Scaling Paradigm
Why ultra‑long context processing stalls and how DeepSeek‑V4 tackles it.
Reasoning models such as DeepSeek‑AI (2025) and OpenAI (2024c) have ushered in a test‑time scaling paradigm that yields large performance gains for LLMs. However, the quadratic cost of vanilla attention (Vaswani et al., 2017) becomes a prohibitive bottleneck when sequences grow to millions of tokens.
Long‑horizon scenarios—complex agentic workflows and massive cross‑document analysis—now demand efficient ultra‑long context support, yet existing open‑source efforts (Bai et al., 2025a; DeepSeek‑AI 2024; MiniMax 2025; Qwen 2025) still inherit this architectural inefficiency.
To break this barrier we introduce the DeepSeek‑V4 series, including preview models DeepSeek‑V4‑Pro (49 B activated) and DeepSeek‑V4‑Flash (13 B activated), which enable efficient processing of one‑million‑token contexts.
Compared with DeepSeek‑V3.2, the new series preserves the DeepSeekMoE framework and Multi‑Token Prediction strategy while adding a hybrid attention mechanism that fuses Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA).
CSA first compresses the key‑value caches along the sequence dimension and then runs DeepSeek Sparse Attention; HCA applies a more aggressive compression to the caches but retains dense attention for critical regions.
We also incorporate Manifold‑Constrained Hyper‑Connections (mHC) to strengthen residual pathways and adopt the Muon optimizer, which speeds convergence and improves training stability.
Infrastructure upgrades include a single fused kernel that fully overlaps MoE computation, communication, and memory access, plus TileLang—a DSL that balances developer productivity with runtime efficiency.
Additional system‑level improvements are batch‑invariant deterministic kernels, tensor‑level checkpointing for fine‑grained recomputation, a hybrid ZeRO strategy for the Muon optimizer, cost‑effective mHC via recomputation, and a two‑stage contextual parallelism scheme.
For inference we design a heterogeneous KV cache with on‑disk storage to enable shared‑prefix reuse, and we apply FP4 quantization‑aware training to MoE expert weights and the QK indexer, reducing memory and compute.
In a 1 M‑token setting DeepSeek‑V4‑Pro consumes only 27 % of the single‑token FLOPs and 10 % of the KV cache size of DeepSeek‑V3.2; DeepSeek‑V4‑Flash pushes this further to 10 % FLOPs and 7 % KV cache.
Although FP4 × FP8 operations currently match FP8 × FP8 peak FLOPs on existing hardware, they could become one‑third more efficient on future accelerators.
Pre‑training consumes 32 T tokens for Flash and 33 T tokens for Pro, after which both models natively support 1 M‑token contexts.
The post‑training pipeline first cultivates domain‑specific experts via Supervised Fine‑Tuning, then refines them with Reinforcement Learning using Group Relative Policy Optimization and a reverse KL loss, before unifying all experts through on‑policy distillation.
On knowledge benchmarks (SimpleQA, Chinese‑SimpleQA, MMLU‑Pro, HLE, GPQA) DeepSeek‑V4‑Pro‑Max narrows the gap to the leading proprietary Gemini‑3.1‑Pro model.
Reasoning evaluations place DeepSeek‑V4‑Pro‑Max ahead of GPT‑5.2 and Gemini‑3.0‑Pro, though it trails GPT‑5.4 and Gemini‑3.1‑Pro by a modest margin.
In agent tasks DeepSeek‑V4‑Pro‑Max matches open‑source leaders Kimi‑K2.6 and GLM‑5.1, while still lagging behind Claude Sonnet 4.5 and Opus 4.5.
For long‑context workloads (1 M‑token) DeepSeek‑V4‑Pro‑Max outperforms Gemini‑3.1‑Pro on academic benchmarks, demonstrating the practical impact of the efficiency gains.
DeepSeek‑V4‑Flash‑Max, while slightly weaker on knowledge tasks due to its smaller scale, attains comparable reasoning results when allocated a larger thinking budget.
Compressed Sparse Attention
We introduce Compressed Sparse Attention, the key trick that makes million‑token context feasible.
Long‑range attention scales quadratically, so processing a million‑token context quickly exhausts memory and compute. The authors therefore replace the dense attention kernel with a two‑stage compression‑then‑sparse scheme.
CSA first shrinks the KV cache by merging every $m$ tokens into a single compressed entry, then lets each query attend only to the top‑$k$ compressed entries – like a librarian who first groups books by genre (compression) and then scans only the relevant genre shelves (sparse attention).
Group tokens (0,1), (2,3), (4,5), (6,7). For each group compute $\mathbf{C}_a = \mathbf{H}\mathbf{W}_a^{KV}$ and $\mathbf{Z}_a = \mathbf{H}\mathbf{W}_a^{Z}$ (using toy weight matrices).
Apply element‑wise multiplication $\mathbf{Z}_a\odot\mathbf{C}_a$, add bias $\mathbf{B}_a$, and row‑softmax across the two tokens in the group to obtain one compressed entry per group.
Result: a compressed KV matrix $\mathbf{C}_{\text{Comp}}$ of shape $4\times2$ (four groups, two‑dimensional entries).
Run the Lightning Indexer on the query token $t=5$: compress the query in the same way, compute scores against the four compressed KV entries, and keep the top‑$k=2$ entries.
Attend only to those two entries, producing the final attention output for token 5.
Compression cuts the KV memory by $m=2$ while the top‑$k$ selector guarantees that each query still sees the most relevant compressed entries, preserving long‑range information with far less cost.
How is CSA different from ordinary sparse attention that simply drops tokens?
Ordinary sparse attention chooses which *original* tokens to attend to, still handling $O(n)$ KV entries. CSA first *compresses* $m$ tokens into a single learned representation, so the attention matrix operates on $O(n/m)$ entries before sparsity is applied. The compression step preserves information from all tokens, whereas plain sparsity discards the unselected ones entirely.
**Figure 2 |** Overall architecture of DeepSeek-V4 series. We use hybrid CSA (Compressed Sparse Attention) and HCA (Heavily Compressed Attention) for attention layers, DeepSeekMoE for feed-forward layers, and strengthen conventional residual connections with mHC.
**Figure 3.** Core architectures of CSA. It compresses the number of KV entries to $\frac{1}{m}$ times, and then applies DeepSeek Sparse Attention for further acceleration. Additionally, a small set of sliding window KV entries is combined with the selected compressed KV entries to enhance local fine-grained dependencies.
mHC replaces the unconstrained residual matrix $\mathbf{B}_\ell$ with a doubly‑stochastic matrix, guaranteeing that each row and column sums to 1 and that the transformation never amplifies signals.
Heavily Compressed Attention
Heavily Compressed Attention compresses the KV cache further to keep long‑context inference tractable.
Long‑context inference is crippled by the quadratic KV cache growth of vanilla attention; even a million‑token window would exhaust GPU memory.
HCA squeezes many consecutive KV entries into a single representative entry, discarding the fine‑grained redundancy that long contexts contain while still keeping a tiny sliding‑window cache for local detail.
How does HCA differ from the earlier Compressed Sparse Attention (CSA) that the paper introduced?
CSA still keeps a sparse index over many small compressed blocks (rate $m$) and selects a top‑$k$ subset at runtime. HCA abandons the sparse index entirely, using a **much larger** compression factor $m'$ to collapse each block into a single entry, and relies on a tiny sliding‑window cache for local detail instead of a learned sparse pattern.
Compute softmax over $Z+B$: $S = \text{softmax}([2.0,1.0,0.5,0.0]) = [0.49,\,0.30,\,0.15,\,0.06]$.
Form the weighted sum: $C_{\text{Comp}} = 0.49\cdot(1,0) + 0.30\cdot(0,1) + 0.15\cdot(1,1) + 0.06\cdot(0,0) = (0.64,\,0.45)$.
The block of four tokens is now represented by the single compressed entry $(0.64,\,0.45)$, reducing the KV length from 4 to 1.
Because the softmax normalizes across the block, the compressed entry always lies inside the convex hull of the original vectors, guaranteeing that no extreme outlier can dominate the representation.
After compression, HCA feeds the compact KV set into a shared‑key‑value MQA layer and applies a grouped output projection that splits the $n_h$ head outputs into $g$ groups before the final $d$‑dimensional projection.
**Figure 4.** Core architectures of HCA. It performs heavier compression, where the KV entries of $m' (\gg m)$ tokens will be consolidated into one. Also, we additionally introduce a small set of sliding window KV entries to enhance local fine-grained dependencies.
Expert Parallelism Optimization
We detail the fine‑grained expert parallelism that overlaps communication and computation to cut latency.
Expert Parallelism (EP) can accelerate Mixture‑of‑Experts models, but its inter‑node communication often dominates latency.
Instead of sending all expert data at once, we split experts into small “waves” and overlap each wave’s communication with the next wave’s computation, so the network stays busy while the GPU works.
How does this fine‑grained EP differ from the naïve “dispatch‑then‑compute” approach?
Naïve EP waits for all experts to finish dispatch before any computation starts, creating a long idle period for the GPU. Fine‑grained EP interleaves dispatch and compute at the wave level, so the GPU never stalls and the network traffic is spread evenly across the layer.
Wave 0 dispatches its tokens to E₀/E₁ (takes 2 µs).
Immediately after dispatch, Wave 0 starts Linear‑1 computation (3 µs).
While Wave 0 computes, Wave 1 begins its dispatch (2 µs) in parallel.
When Wave 0 finishes Linear‑1, it proceeds to Linear‑2 while Wave 1 is still dispatching.
After both waves finish their computations, their results are combined in a single Reduce step (1 µs).
The pipeline keeps both network and compute resources busy, turning a serial latency of ≈ 10 µs into an overlapped latency of ≈ 5 µs.
Profiling shows that within a MoE layer the total communication time is already smaller than the compute time, so hiding it behind computation yields negligible performance loss even on modest interconnects.
For DeepSeek‑V4‑Pro each token‑expert pair performs 6 $d$ FLOPs but moves only 3 $d$ bytes, giving $V_{\text{comp}}/V_{\text{comm}} = 2d = 6144$ FLOPs/Byte. Thus 1 GB/s of bandwidth suffices for 6.1 TFLOP/s of compute.
**Figure 5.** Illustration of our EP scheme with related works. Comet (Zhang et al., 2025b) overlaps Dispatch with Linear-1, and Linear-2 with Combine, separately. Our EP scheme achieves a finer-grained overlapping by splitting and scheduling experts into waves. The theoretical speedup is evaluated in the configuration of the DeepSeek-V4-Flash architecture.
Beyond EP, we built a DSL called TileLang to author fused kernels quickly, and we introduced Host Codegen to push per‑invocation checks into generated C++ launchers, cutting CPU‑side overhead from ~100 µs to <1 µs.
Host‑generated launcher for a fused TileLang kernel (simplified).
We also integrated the Z3 SMT solver to prove integer constraints during compilation, enabling aggressive vectorization without sacrificing correctness.
Our kernels preserve bitwise reproducibility by disabling fast‑math by default and exposing IEEE‑compliant intrinsics for users who need strict IEEE‑754 behavior.
For deterministic training we introduced a dual‑kernel batch‑invariant decoding strategy: a single‑SM kernel handles full waves, while a multi‑SM kernel finishes the trailing partial wave, both following identical accumulation order.
**Require:** Learning rate $\eta$, momentum $\mu$, weight decay $\lambda$, update rescaling factor $\gamma$ 1: **for** each training step $t$ **do** 2: **for** each logically independent weight $W \in \mathbb{R}^{n \times m}$ **do** 3: $G_t = \nabla_W \mathcal{L}_t(W_{t-1})$ $\quad \triangleright$ Compute gradients 4: $M_t = \mu M_{t-1} + G_t$ $\quad \triangleright$ Accumulate momentum buffer 5: $O'_t = \text{HybridNewtonSchulz}(\mu M_t + G_t)$ $\quad \triangleright$ Nesterov trick and hybrid Newton-Schulz 6: $O_t = O'_t \cdot \sqrt{\max(n, m)} \cdot \gamma$ $\quad \triangleright$ Rescale the update RMS 7: $W_t = W_{t-1} \cdot (1 - \eta\lambda) - \eta O_t$ $\quad \triangleright$ Perform weight decay and update 8: **end for** 9: **end for**
Training Determinism and Cache Layout
Deterministic training and memory‑efficient infrastructure enable stable, high‑throughput DeepSeek‑V4 execution.
Training at million‑token scale is fragile: nondeterministic floating‑point accumulation can hide bugs and cause loss spikes, while memory‑heavy kernels threaten throughput.
Instead of letting all SMs atomically add to a shared gradient buffer (which yields nondeterministic sums), each SM writes to its own private buffer and we later sum those buffers in a fixed order.
SM 0 writes its three values into its private buffer.
SM 1 writes its three values into its private buffer.
The deterministic reducer adds SM 0’s buffer first, then SM 1’s, yielding the final sum [0.5, 0.3, 0.5].
Even though the total sum is the same as a naïve atomicAdd, the exact binary representation is fixed, enabling reproducible debugging.
Why not just keep using atomicAdd if the final sum is mathematically identical?
AtomicAdd schedules the adds in an order that depends on thread scheduling and hardware timing; because floating‑point addition is not associative, different orders produce slightly different rounding errors, which can accumulate into noticeable loss spikes.
Each rank first sorts its incoming tokens into a fixed order before any cross‑rank writes, then isolates buffers per rank so that writes never interleave.
Each rank sorts its IDs ascending, yielding the same order as listed.
Rank 0 writes its two tokens into buffer 0, rank 1 into buffer 1, rank 2 into buffer 2.
The receiver concatenates buffers in rank order, producing the deterministic sequence [0,3,1,4,2,5].
Because the ordering rule is identical on all ranks, the final concatenated buffer is independent of network timing.
How does this differ from the usual all‑reduce of MoE gradients?
All‑reduce mixes contributions from all ranks in a tree or ring, where the reduction order varies with network latency; our scheme fixes the order by pre‑sorting and using isolated buffers, so the same binary result is produced every run.
When the output dimension is tiny (e.g., 24), we split the K‑dimension across SMs; each SM writes its partial result to a separate output slice, then a second kernel deterministically adds the slices.
Each SM writes its vector to buffer i.
The deterministic reducer adds buffers in order 0→1→2→3, yielding the final vector $v$ = p₀ + p₁ + p₂ + p₃.
Because addition order is fixed, the exact floating‑point rounding of each element is identical across runs.
Why not use a parallel reduction tree for the split‑k outputs?
A parallel tree would combine partial results in a nondeterministic order depending on thread scheduling, re‑introducing the same rounding‑error variability we are trying to eliminate.
Muon needs the full gradient matrix, which conflicts with ZeRO’s per‑parameter sharding; we therefore assign whole parameter matrices to ranks using a knapsack‑style load balancer and pad each bucket to the size of the largest one.
Knapsack assigns matrices 120 MiB + 30 MiB to Rank 0 (total 150 MiB) and 80 MiB + 50 MiB to Rank 1 (total 130 MiB).
The larger bucket (Rank 0) is 150 MiB; Rank 1 pads its bucket with 20 MiB of dummy data.
Both ranks now perform a single reduce‑scatter on 150 MiB, keeping communication balanced.
Padding equalizes communication volume without significantly increasing memory, preserving ZeRO’s efficiency.
Why not simply shard each matrix element‑wise as ZeRO does for AdamW?
Muon’s update requires the full matrix to compute a Newton‑Schulz iteration; splitting the matrix would break the algorithm because the iteration needs the entire dense tensor on a single device.
We first quantize MoE gradients to BF16 using stochastic rounding (preserving unbiasedness), then exchange them with an all‑to‑all, finally sum locally in FP32 to avoid accumulation error.
Why not keep the gradients in FP32 throughout the all‑reduce?
Transmitting FP32 doubles the bandwidth requirement; stochastic‑rounded BF16 cuts the traffic in half while the subsequent FP32 sum restores exactness.
We fuse the forward and backward passes of mHC into a single kernel and checkpoint only the most expensive intermediate tensors, recomputing the cheap ones on demand.
Why not checkpoint every activation to avoid any recomputation?
Full checkpointing would require storing all intermediate tensors, which would exceed GPU memory for million‑token contexts; selective recomputation strikes a better memory‑compute trade‑off.
Ranks first exchange the last m uncompressed KV entries, then each rank compresses locally and finally performs an all‑gather of the compressed pieces.
How does this differ from naïve context parallelism that simply shards the sequence?
Naïve sharding would require each rank to compress across shard boundaries, which is impossible without exchanging uncompressed tokens; our two‑stage scheme guarantees that every rank has the necessary raw tokens before compression.
We separate volatile sliding‑window attention (SWA) entries from uncompressed tail tokens, storing them in a fixed‑size per‑request block that can be quickly indexed.
CSA produces k₁ = 12/4 = 3 compressed tokens.
HCA produces k₂ = 12/6 = 2 compressed tokens.
The block stores these 5 compressed entries plus the SWA segment for the last n₍win₎ = 4 tokens.
The block size is independent of the absolute sequence length, allowing uniform kernel launches across layers.
Why not merge the SWA and compression segments into a single contiguous buffer?
Separating them lets the kernel apply different eviction policies: SWA entries are overwritten in a sliding window, while compressed entries are immutable until the next compression step, reducing synchronization overhead.
Depending on deployment constraints we either store compressed KV entries, checkpoint SWA periodically, or recompute SWA on‑the‑fly, trading storage for compute.
When would the Zero SWA Caching strategy be preferable?
In environments where SSD write endurance is a concern or where storage cost dominates, accepting extra recomputation saves disk I/O while still delivering low‑latency responses thanks to the fast BF16 kernels.
**Figure 6.** Illustration of the KV cache Layout for DeepSeek-V4. The KV cache is organized into two primary components: a classical KV cache for CSA/HCA, and a state cache for SWA and unready-for-compression tokens in CSA/HCA. In the state cache, each request is assigned a fixed-size cache block. Within this block, the SWA segment stores the KV entries corresponding to the most recent $n_{\text{win}}$ tokens, while the CSA/HCA segment stores uncompressed tail states that are not yet ready for compression. In the classical KV cache, we allocate multiple blocks per request. Each cache block covers $\text{lcm}(m, m')$ original tokens, producing $k_1 = \frac{\text{lcm}(m, m')}{m}$ CSA compressed tokens and $k_2 = \frac{\text{lcm}(m, m')}{m'}$ HCA compressed tokens.
Data Construction
Details of the pre‑training corpus and model configurations for DeepSeek‑V4.
We expand the DeepSeek‑V3 corpus by adding longer, higher‑quality documents and by filtering out auto‑generated web content, which reduces the risk of model collapse. The resulting pre‑training set exceeds 32 T tokens and includes mathematics, code, multilingual web pages, and long‑form scientific material.
Tokenization builds on the DeepSeek‑V3 tokenizer but introduces a handful of special tokens for context construction while keeping the vocabulary at 128 K. We retain the token‑splitting and Fill‑in‑Middle (FIM) strategies, and we pack documents from heterogeneous sources into sequences to minimise truncation, applying sample‑level attention masking during pre‑training.
DeepSeek‑V4‑Flash uses 43 Transformer layers with hidden dimension $d$ = 4096. The first two layers employ pure sliding‑window attention; subsequent layers interleave Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA). CSA compresses queries at rate $m$ = 4, uses $n_I$ = 64 indexer heads of dimension $c_I$ = 128, and selects $k$ = 512 KV entries for sparse attention. HCA compresses at rate $m'$ = 128, with $n_h$ = 64 query heads of dimension $c$ = 512 and compression dimension $d_c$ = 1024.
DeepSeek‑V4‑Pro expands to 61 layers with hidden dimension $d$ = 7168. Like Flash, the first two layers use HCA, and later layers interleave CSA and HCA. CSA here uses $m$ = 4, $n_h$ = 64 query heads, $c_I$ = 128, and $k$ = 1024 KV entries; HCA retains $m'$ = 128 with $n_h$ = 128 query heads, $c$ = 512, and $d_c$ = 1536. Both variants employ MoE layers with Hash routing for the first three MoE blocks, 1 shared expert plus 256 (Flash) or 384 (Pro) routed experts, each expert having an intermediate hidden dimension of 2048 (Flash) or 3072 (Pro), and six experts activated per token.