DeepSeek-V3 Technical Report
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang
DeepSeek-V3 is a 671B-parameter Mixture-of-Experts model achieving state-of-the-art performance at a fraction of typical training costs.
How can we scale a Mixture-of-Experts language model to 671B parameters while maintaining efficient inference and training costs?
Large-scale language models are increasingly expensive to train, often requiring massive compute clusters that struggle with communication bottlenecks and inefficient load balancing. DeepSeek-V3 addresses this by combining a Mixture-of-Experts (MoE) architecture with a novel auxiliary-loss-free load balancing strategy and an FP8 mixed-precision training framework. The model achieves performance comparable to leading closed-source models like GPT-4o while requiring only 2.788 million H800 GPU hours for full training.
Paper Primer
The core innovation is the co-design of algorithms, hardware, and training frameworks to maximize efficiency. The model uses Multi-head Latent Attention (MLA) to compress the key-value cache and a Multi-Token Prediction (MTP) objective to densify training signals, effectively forcing the model to plan representations for future tokens.
To handle the communication overhead of cross-node MoE training, the authors developed the DualPipe algorithm: it overlaps computation and communication phases to hide latency, ensuring near-zero all-to-all communication overhead even as the model scales.
DeepSeek-V3 achieves state-of-the-art performance on coding and math benchmarks among open-source models.
The model scored 88.5 on MMLU and 75.9 on MMLU-Pro, outperforming other open-source models and rivaling closed-source benchmarks. Training required only 2.788M H800 GPU hours, a significant reduction in cost compared to models of similar scale.
The FP8 mixed-precision training framework maintains numerical stability without performance loss.
Relative loss error compared to a BF16 baseline remained consistently below 0.25% throughout training. The framework enables training on extremely large-scale models while reducing both memory footprint and communication overhead.
Why does the model use an auxiliary-loss-free strategy for load balancing?
Conventional auxiliary losses used to force load balancing often impair overall model performance; this strategy uses a dynamic bias term to balance expert load without the performance degradation associated with explicit loss penalties.
What is the primary purpose of the Multi-Token Prediction (MTP) objective?
MTP densifies training signals by predicting multiple future tokens at each position, which improves data efficiency and encourages the model to pre-plan representations for better future token prediction.
Introduction and Overview
DeepSeek‑V3 pushes open‑source MoE models toward higher performance while keeping training and inference costs low.
Current open‑source LLMs still lag behind closed‑source leaders, especially when scaling Mixture‑of‑Experts (MoE) models; the dominant bottlenecks are memory‑intensive KV caches and costly expert‑load balancing, which together inflate training time and inference latency.
We need a way to keep MoE models fast and cheap without sacrificing the quality gains that come from activating many experts per token.
DeepSeek‑V3 addresses this gap by combining several orthogonal techniques: Multi‑head Latent Attention (MLA) compresses the KV cache, an auxiliary‑loss‑free load‑balancing scheme removes performance‑harming regularizers, Multi‑Token Prediction (MTP) improves training efficiency, FP8 precision cuts compute and memory, and DualPipe eliminates pipeline stalls.
The model is trained on 14.8 T tokens using only 2.788 M H800 GPU‑hours (≈ \$5.576 M), and the entire pre‑training run completed without any irrecoverable loss spikes or rollbacks, demonstrating that the engineering choices yield a stable, cost‑effective training pipeline.
**Figure 1.** Benchmark performance of DeepSeek-V3 and its counterparts.
The core trade‑off in MoE systems—balancing expert activation cost against performance gain—is resolved in DeepSeek‑V3 by compressing attention, removing auxiliary loss penalties, and leveraging low‑precision training.
Core Architecture: MLA and MoE
Describes the core MLA and DeepSeekMoE mechanisms that cut memory and balance expert load.
Standard multi‑head attention stores a full key‑value (KV) cache for every token, which quickly exhausts GPU memory during long generation. The authors therefore replace the cache with a compact latent representation (MLA) and restructure the feed‑forward layer as a mixture‑of‑experts (DeepSeekMoE) that self‑balances without an auxiliary loss.
MLA compresses the key and value tensors into a low‑rank latent space before caching, so the memory needed for the KV cache shrinks dramatically while the attention computation still sees the full information after up‑projection. It works like summarizing a library into a concise index — the index fits in memory yet still lets you locate any book quickly.
Form $KV^t=[k_{t,1};k_{t,2};v_{t,1};v_{t,2}]$ (8 numbers).
Down‑project: $KV_c^t = W_{DKV}\,KV^t$ yields a $2$‑dimensional latent vector.
Cache only $KV_c^t$ (2 numbers) instead of the original 8.
When attention is needed, up‑project: $\hat{K}^t = W_{U K}\,KV_c^t$, $\hat{V}^t = W_{U V}\,KV_c^t$, each producing 4‑dimensional reconstructed keys/values.
Proceed with standard scaled‑dot‑product attention using $\hat{K}^t,\hat{V}^t$ and the up‑projected queries.
The cache size drops from $8$ numbers to $2$, a $4\times$ memory saving, while the up‑projection restores the full $8$‑dimensional representation for accurate attention.
How does MLA differ from ordinary multi‑head attention?
Ordinary attention stores the full $K$ and $V$ tensors for every token, which grows linearly with sequence length. MLA first compresses $K$ and $V$ into a tiny latent vector, caches only that, and reconstructs the full tensors on‑the‑fly with learned up‑projections. The extra linear layers add negligible compute but cut the KV memory dramatically.
Even with a compact KV cache, the feed‑forward stage can dominate compute if every token routes to many experts. DeepSeekMoE solves this by mixing shared experts with routed experts and by balancing load without an auxiliary loss.
DeepSeekMoE partitions the FFN into many small expert networks; each token is sent to a few routed experts while also passing through a set of shared experts. A bias term added to the routing scores automatically steers tokens toward under‑used experts, eliminating the need for an explicit auxiliary loss.
Compute biased scores: token 1 → $[0.6+0,\,0.4+0]=[0.6,0.4]$, token 2 → $[0.3+0,\,0.7+0]=[0.3,0.7]$.
Select top‑1 routed expert per token: token 1 picks expert 1, token 2 picks expert 2.
Observe load: each routed expert receives one token, so load is balanced; no bias update needed.
Now imagine a third token with affinities $s_{1,3}=0.9$, $s_{2,3}=0.2$. Biased scores become $[0.9,0.2]$, so expert 1 would get a second token.
Because expert 1 is now overloaded, the training rule decreases $b_1$ (e.g., $b_1\gets b_1-\gamma$) and increases $b_2$, making the next token more likely to select expert 2.
The bias term dynamically steers traffic away from hot experts, achieving load balance without an extra loss term.
Why can DeepSeekMoE avoid the auxiliary loss that other MoE models need?
Instead of penalizing imbalance with a separate loss, the model directly modifies the routing scores via a per‑expert bias $b_i$. The bias is increased for under‑used experts and decreased for overloaded ones, so the routing decision itself enforces balance. Because the bias is part of the routing computation, no extra gradient term is required.
**Figure 2.** Illustration of the basic architecture of DeepSeek-V3. Following DeepSeek-V2, we adopt MLA and DeepSeekMoE for efficient inference and economical training.
Multi-Token Prediction Objective
Multi-Token Prediction densifies training signals by sequentially forecasting several future tokens.
Training large language models suffers from sparse supervision at each position, limiting data efficiency. Extending the prediction horizon to multiple future tokens densifies the learning signal and lets the model pre‑plan its representations. Our Multi-Token Prediction (MTP) objective implements this idea while preserving causality.
Instead of guessing the next word one step at a time, MTP asks the model to write the next few words in order, like a writer sketching the next two sentences before committing to the first.
How does this sequential MTP differ from the parallel multi‑head variant used by Gloeckle et al. (2024)?
In the parallel variant each future token is predicted from the same current representation, so the predictions are independent and the causal chain is broken. Our sequential design feeds the output of depth $k\!-\!1$ into depth $k$, so the $k$‑th prediction is conditioned on all earlier predictions, preserving the full causal dependency.
Depth 1, $i=1$: concatenate $h_{0,1}$ (main‑model hidden state) with $Emb(t_{2})$, apply identity RMSNorm, then $M_1$ drops the second half, yielding $h'_{1,1}$.
Feed $h'_{1,1}$ into $\text{TRM}_1$ to obtain $h_{1,1}$.
OutHead maps $h_{1,1}$ to logits; softmax gives $P_{1,3}$, the distribution for token $t_3$.
Depth 2, $i=1$: concatenate $h_{1,1}$ with $Emb(t_{3})$, identity RMSNorm, project with $M_2$ to get $h'_{2,1}$.
Process $h'_{2,1}$ through $\text{TRM}_2$ → $h_{2,1}$, then OutHead produces $P_{2,4}$ for token $t_4$.
The second‑step prediction is conditioned on the representation already shaped by the first prediction, so the model learns to plan ahead while respecting causality.
Obtain the current hidden state $h_{k-1,i}$ (or the main‑model state for $k=1$).
Embed the future token $t_{i+k}$ with the shared embedding layer.
Apply RMSNorm to both vectors.
Concatenate the two normalized vectors and linearly project with $M_k$ to produce $h'_{k,i}$.
Pass $h'_{k,i}$ through the depth‑specific Transformer block $\text{TRM}_k$ to obtain $h_{k,i}$.
Feed $h_{k,i}$ to the shared output head, apply softmax, and obtain the probability distribution $P_{k,i+k+1}$ for the $k$‑th future token.
**Figure 3.** Illustration of our Multi-Token Prediction (MTP) implementation. We keep the complete causal chain for the prediction of each token at each depth.
DualPipe: Communication-Efficient Pipeline
Efficient pipeline parallelism is achieved by overlapping computation and communication with DualPipe.
DeepSeek‑V3 runs on a 2048‑GPU H800 cluster (8 GPUs per node, NVLink/NVSwitch intra‑node, InfiniBand inter‑node). The HAI‑LLM framework orchestrates 16‑way pipeline, 64‑way expert, and ZeRO‑1 data parallelism, providing the substrate for the novel DualPipe scheduling.
DualPipe stitches forward and backward chunks together so that communication can hide behind useful computation, eliminating idle “pipeline bubbles”.
How does DualPipe differ from the classic 1F1B pipeline parallelism?
1F1B runs a full forward chunk (F) then a full backward chunk (B), leaving a bubble of $(PP\!-\!1)(F\!+\!B)$. DualPipe interleaves forward and backward sub‑chunks (F & B) and overlaps all‑to‑all communication, shrinking the bubble term to $\bigl(\frac{PP}{2}\!-\!1\bigr)(F\&B\!+\!B\!-\!3W)$ and thus eliminating most idle time.
Time 0: Rank 0 starts forward sub‑chunk F₀; Rank 3 starts backward‑input sub‑chunk `B_in`₃.
Time 1: While F₀ computes attention, Rank 0 initiates dispatch D₀ to send its activations to Rank 1.
Time 2: Rank 1 receives D₀, begins its own forward sub‑chunk F₁, and simultaneously runs combine C₃ for the backward data arriving from Rank 2.
Time 3: All ranks have both a compute sub‑chunk and a communication sub‑chunk active; no rank is idle.
Time 4: The last backward‑input sub‑chunk `B_in`₀ finishes, and the final combine C₀ merges the remaining gradients.
By overlapping a forward sub‑chunk with a backward‑input sub‑chunk and hiding all‑to‑all traffic inside those windows, the schedule achieves near‑zero idle time even with only two micro‑batches.
Split each pipeline stage into four components: attention, all‑to‑all dispatch, MLP, all‑to‑all combine.
For backward stages, further split attention and MLP into input‑wise and weight‑wise halves.
Launch a forward sub‑chunk on the leftmost rank while simultaneously launching a backward‑input sub‑chunk on the rightmost rank.
As soon as a sub‑chunk finishes its compute part, start the corresponding all‑to‑all communication (dispatch or combine) without waiting for the opposite direction.
Continue feeding micro‑batches from both ends, keeping every rank occupied with either compute or communication at each timestep.
Terminate when the last backward‑weight sub‑chunk completes and its combine finishes.
**Figure 4.** Overlapping strategy for a pair of individual forward and backward chunks (the boundaries of the transformer blocks are not aligned). Orange denotes forward, green denotes "backward for input", blue denotes "backward for weights", purple denotes PP communication, and red denotes barriers. Both all-to-all and PP communication can be fully hidden.
**Figure 5.** Example DualPipe scheduling for 8 PP ranks and 20 micro-batches in two directions. The micro-batches in the reverse direction are symmetric to those in the forward direction, so we omit their batch ID for illustration simplicity. Two cells enclosed by a shared black border have mutually overlapped computation and communication.
**Table 2.** Comparison of pipeline bubbles and memory usage across different pipeline parallel methods. F denotes the execution time of a forward chunk, B denotes the execution time of a full backward chunk, W denotes the execution time of a "backward for weights" chunk, and F&B denotes the execution time of two mutually overlapped forward and backward chunks.
FP8 Training and Implementation
We detail cross‑node communication and FP8 mixed‑precision tricks that cut memory and speed training.
Training a massive MoE model strains both inter‑GPU bandwidth and GPU memory. The authors therefore redesign communication and precision to keep the system efficient.
Instead of letting every token wander through all GPUs, each token is sent to at most four nodes and then instantly forwarded inside the node, so the slow IB link carries only a few copies while the fast NVLink finishes the job.
Step 1 – IB dispatch: the token is sent from Node A‑GPU 0 to Node B‑GPU 0 (the only IB hop).
Step 2 – NVLink forward: on Node B, a warp immediately forwards the token from GPU 0 to GPU 1 via NVLink because GPU 1 also hosts a needed expert.
Step 3 – NVLink receive: Node A‑GPU 1 receives its local expert copy directly over NVLink (no IB involved).
Step 4 – Combine: each GPU runs a warp that accumulates the partial results from its local experts, overlapping with the next compute layer.
Limiting the IB hop to a single copy per token eliminates the quadratic traffic growth that would otherwise dominate as the number of experts rises.
How does this differ from a standard all‑reduce across all GPUs?
All‑reduce forces every GPU to exchange data with every other GPU, incurring O(N²) traffic. The presented scheme restricts IB traffic to one copy per token and uses fast NVLink only for intra‑node fan‑out, turning the pattern into O(N) IB traffic plus cheap local broadcasts.
Instead of keeping every intermediate activation in GPU memory, the system recomputes cheap layers on the backward pass and stores only the most expensive pieces in low‑precision buffers.
Forward: compute RMSNorm → produce tensor $A$ (2 × 4 × 8) and immediately discard $A$ after feeding it to the next layer.
Backward: when gradients reach RMSNorm, recompute $A$ from the saved inputs (the same cheap linear ops) and use it to back‑propagate.
MLA up‑projection (dimension 8 → 16) is similarly recomputed on the fly during the gradient pass.
Because RMSNorm and MLA up‑projection involve only a few matrix‑vector products, the extra compute is far cheaper than the memory saved by not storing their large activation maps.
Why not keep the RMSNorm activations and avoid recomputation?
Storing the (2 × 4 × 8) tensor for every layer would multiply memory usage by the number of layers, quickly exceeding GPU capacity for deep MoE models. The recompute cost is negligible compared with the memory bottleneck.
Think of FP8 as a low‑resolution sketch that captures the overall shape, while a higher‑precision accumulator refines the details during the heavy matrix multiply.
Compute $s = \max(|-0.8|,|0.6|)=0.8$.
Derive scaling factor $ \alpha = 2^{\lceil\log_2 s\rceil}=1$ (integral power of 2).
Divide each element by $\alpha$ (no change) and round to the nearest representable E4M3 value, yielding a quantized tile $Q$.
During GEMM, multiply $Q$ with a weight block (128 × 128) also quantized similarly; after $N_c=128$ multiplications, promote the partial sum to FP32 and continue accumulating.
The per‑tile scaling adapts to local outliers, so a single large value does not dominate the entire tensor’s exponent, preserving precision where it matters.
Why not keep the whole model in BF16 instead of mixing FP8 and higher precisions?
BF16 still stores 16 bits per value, so memory and bandwidth savings are modest. FP8 cuts those costs by half, but naïve FP8 would overflow on outliers. The fine‑grained scaling and FP32 accumulation restore the missing dynamic range, giving BF16‑level accuracy with FP8’s efficiency.
**Figure 6.** The overall mixed precision framework with FP8 data format. For clarification, only the Linear operator is illustrated.
**Table 1.** Training costs of DeepSeek-V3, assuming the rental price of H800 is \$2 per GPU hour.
Pre-Training Data and Strategy
Pre‑training scales to 671 B parameters, consumes 14.8 T tokens, and reshapes data composition from V2 to V3.
DeepSeek‑V3 scales to 671 B total parameters while activating only 37 B per token.
Model architecture combines 61 layers, MLA with 128 heads, and MoE layers with 256 routed experts.
Compared with DeepSeek‑V2, which used roughly 12 T tokens and a 400 B‑parameter model, V3 expands both data volume and model capacity, enabling stronger multilingual and code‑understanding abilities.
A tokenizer that breaks text into byte‑level pieces, guaranteeing that any Unicode string can be represented without out‑of‑vocabulary tokens.
How does the combined punctuation/line‑break token differ from a regular token?
It encodes a punctuation symbol together with a newline as a single unit, which shortens sequences but can bias the model toward treating the newline as part of the punctuation; random splitting during training forces the model to handle both merged and separate representations.
Model hyper‑parameters include 61 Transformer layers, hidden dimension 7168, and MLA configured with 128 heads ($n_h\!=\!128$) each of dimension $d_h\!=\!128$. KV caches are compressed to $d_c\!=\!512$ and queries to $d'_c\!=\!1536$, while decoupled queries use $d_{R h}\!=\!64$. MoE layers replace most FFNs, featuring 1 shared expert plus 256 routed experts (intermediate size 2048); eight experts are active per token and each token is routed to at most four nodes.
Long‑context capability is added post‑pre‑training via YaRN. Two 1000‑step phases expand the context window from 4 K to 32 K and then to 128 K tokens, using YaRN hyper‑parameters $s\!=\!40$, $\alpha\!=\!1$, $\beta\!=\!32$, and $t\!=\!0.1\ln s\!+\!1$. Both phases run at a learning rate of $7.3\!\times\!10^{-6}$.
**Figure 8.** Evaluation results on the "Needle In A Haystack" (NIAH) tests. DeepSeek-V3 performs well across all context window lengths up to 128K.
Data composition shifts from V2 to V3.
Base Model Performance
DeepSeek‑V3 outperforms peer models on most benchmarks while using far less training compute.
DeepSeek‑V3‑Base wins 8 / 10 evaluated benchmarks, with especially large gaps on code and math tasks.
Table 3 shows DeepSeek‑V3‑Base achieving the highest scores on English, Code, Math, Chinese, and Multilingual benchmarks.
**Table.** Comparison of base model performance across various benchmarks.
Adding the Multi‑Token Prediction (MTP) module consistently raises scores across all benchmarks.
Table 4 reports higher numbers for the “Small MoE w/ MTP” and “Large MoE w/ MTP” rows compared with their baselines.
Removing auxiliary losses and using the auxiliary‑loss‑free balancing strategy yields better performance on most benchmarks.
Table 5 shows higher scores for “Aux‑Loss‑Free” rows compared with “Aux‑Loss‑Based” rows at both small and large scales.
DeepSeek‑V3 delivers state‑of‑the‑art performance on English, code, and math while consuming a fraction of the training compute of competing dense models.
Post-Training and Alignment
Post‑training combines supervised fine‑tuning and a GRPO‑based RL loop.
This section details how DeepSeek‑V3 is refined after the base model: a two‑epoch supervised fine‑tuning (SFT) stage followed by a reinforcement‑learning (RL) stage that uses Group Relative Policy Optimization.
Instead of a separate critic, GRPO treats a batch of sampled outputs as a “group” and uses their relative scores to compute a baseline, letting the policy improve by comparing against its peers.
How does GRPO differ from standard PPO‑style policy gradients?
PPO uses a single baseline (often a value network) and clips the policy ratio per sample. GRPO replaces that baseline with the mean reward of a sampled group, so the advantage is computed relative to peers rather than an external critic, and the KL penalty is applied against a fixed reference model instead of a learned value function.
Reward modeling combines a deterministic rule‑based component for tasks with exact answers (e.g., math boxes, LeetCode test cases) and a learned model‑based component that evaluates free‑form answers using chain‑of‑thought supervision.
Chat Model Evaluation
DeepSeek‑V3 outperforms all baselines on key language and reasoning benchmarks.
DeepSeek‑V3 achieves 89.1 % EM on MMLU‑Redux, a +8.8‑point gain over the strongest DeepSeek‑V2.5 baseline.
Table row “V3” reports 89.1 % on MMLU‑Redux, while DeepSeek‑V2.5‑0905 reports 80.3 %.
Reward and Open-Ended Benchmarks
Open‑ended and reward‑based evaluations show DeepSeek‑V3 leading open‑source models across conversation, coding, and alignment benchmarks.
DeepSeek‑V3 attains an 85.5 % win rate on the Arena‑Hard open‑ended conversation benchmark, the highest among open‑source models and comparable to top closed‑source systems.
Table 7 reports 85.5 % for DeepSeek‑V3 versus 81.2 % for the next best open‑source model.
Beyond the headline numbers, DeepSeek‑V3 achieves roughly an 86 % win rate against the legacy GPT‑4‑0314 baseline on Arena‑Hard, marking the first open‑source model to surpass the 85 % threshold on this challenging benchmark.
On math‑focused benchmarks (AIME 2024, MATH‑500, CNMO 2024) DeepSeek‑V3 outperforms the second‑best Qwen2.5‑72B by about 10 % absolute score, underscoring the impact of the distillation pipeline.
In Chinese factual QA, DeepSeek‑V3 beats Qwen2.5‑72B by 16.4 points on Chinese SimpleQA despite using a smaller pre‑training corpus, highlighting its balanced multilingual training.
**Table.** Comparison of DeepSeek V3 with other state-of-the-art models across various benchmarks.
**Table 7.** English open-ended conversation evaluations. For AlpacaEval 2.0, we use the length-controlled win rate as the metric.
**Table 8.** Performances of GPT-4o, Claude-3.5-sonnet and DeepSeek-V3 on RewardBench.
Table 9 quantifies the benefit of the DeepSeek‑R1 distillation pipeline: LiveCodeBench pass@1 rises from 31.1 % to 37.4 % and MATH‑500 pass@1 from 74.6 % to 83.2 % when the distillation step is added.
Ablation Studies
We revisit the key tricks—MLA, MTP, and balancing—and examine how each ablation impacts performance.
The paper’s core premise is that DeepSeek‑V3 replaces standard attention with Multi‑head Latent Attention, adds Multi‑Token Prediction, and adopts a custom communication‑efficient pipeline (DualPipe) while training in FP8 precision.
Instead of adding an auxiliary loss to steer expert activation, the model directly equalizes load across experts using a deterministic statistic, eliminating the extra loss term.
How does auxiliary‑loss‑free balancing differ from the traditional auxiliary‑loss‑based approach?
The traditional method adds a separate loss term that penalizes deviation from a target load, which introduces extra gradients and hyper‑parameters. The auxiliary‑loss‑free variant removes that term entirely and instead rescales the routing logits directly, achieving balance deterministically and with lower memory overhead.
Distillation from DeepSeek‑R1 was ablated by training a DeepSeek‑V2.5 baseline on short chain‑of‑thought data versus using expert‑generated long‑CoT data. The latter yields sizable gains on LiveCodeBench and MATH‑500, but also lengthens average responses.