Kimi K3: Open Frontier Intelligence

Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, M. C., Jianfeng Cai, Xinyuan Cai, Peizhou Cao, Yuxuan Cao, Ziwei Chai, Y. Charles, H. S. Che, Guanduo Chen, Guangyu Chen, Guanzheng Chen, Huarong Chen, Jia Chen, Jianlong Chen, Jun Chen, Kexin Chen, Peng Chen, Ruijue Chen, Wentao Chen, Xin Chen, Yang Chen, Yanru Chen, Yifei Chen, Yingjiang Chen, Yuankun Chen, Yujie Chen, Yutian Chen, Zhirong Chen, Dazhi Cheng, Yean Cheng, Jialei Cui, Jingbing Cui, Anqi Dai, Jiaqi Deng, Hao Ding, Rui Ding, Shaofeng Ding, Mengfan Dong, Mengnan Dong, Yuhao Dong, Yuxin Dong, Angang Du, Chenzhuang Du, Dikang Du, Jusen Du, Yulun Du, Yu Fan, Jing Feng, Qiulin Feng, Yichen Feng, Kelin Fu, Qiang Fu, Fuxuan Gao, Hongcheng Gao, Jingyue Gao, Tong Gao

Kimi K3 is a 2.8T-parameter multimodal Mixture-of-Experts model scaling pre-training and long-horizon agentic RL to the frontier.

How does Kimi K3 scale multimodal Mixture-of-Experts models to 2.8 trillion parameters while maintaining efficient inference and long-context reasoning?

Open-source models have struggled to scale simultaneously across both pre-training parameter counts and test-time reasoning effort, often lagging behind proprietary systems in long-horizon agentic tasks. Kimi K3 addresses this by combining a 2.8T-parameter Mixture-of-Experts architecture with a million-token context window, utilizing Kimi Delta Attention for efficient sequence mixing and Stable LatentMoE for sparse, stable channel activation. The model achieves frontier-level performance across coding, reasoning, and vision tasks, consistently outperforming other open-source models while trailing only the most powerful proprietary systems.

Paper Primer

Kimi K3 scales information flow through three architectural innovations: Kimi Delta Attention (KDA) for efficient long-sequence mixing, Attention Residuals (AttnRes) for non-sequential depth-wise information access, and Stable LatentMoE for sparse, stable expert routing. These components allow the model to maintain a 1-million-token context window while activating only 16 of 896 experts per token.

The post-training pipeline uses Multi-Teacher On-Policy Distillation (MOPD) to consolidate domain-specific policies—trained via reinforcement learning across coding, general agents, and reasoning—into a single unified model. This process is supported by a unified white-box RL environment that dynamically constructs agent harnesses to prevent overfitting to specific interaction protocols.

Kimi K3 delivers a 2.5× improvement in overall scaling efficiency compared to its predecessor, Kimi K2.

Scaling-law studies conducted on held-out OOD validation data across the new model family.

The model achieves frontier-level performance on long-horizon coding, agentic, and vision tasks.

Extensive benchmark evaluations against both open and proprietary models. Consistently outperforms all evaluated open-source models.

Why does the paper emphasize "test-time scaling" as a distinct axis from pre-training?

The authors argue that while pre-training scales the foundation, test-time scaling—via reinforcement learning and adaptive reasoning budgets—is essential for eliciting sophisticated reasoning and long-horizon agentic behaviors that static pre-training alone cannot provide.

What is the primary trade-off in using Kimi Delta Attention (KDA) instead of standard softmax attention?

KDA replaces the memory-intensive, growing key-value cache of softmax attention with a fixed-size recurrent state, enabling efficient long-context processing at the cost of requiring specialized kernels to manage the serial dependencies of the recurrent updates.

Introduction and Motivation

We frame the need to scale open‑source MoE models both in size and context.

Open‑source large‑scale models have grown to the 1 T‑parameter regime, but the frontier now lies at several‑trillion parameters and million‑token contexts, a gap that threatens to widen as proprietary systems continue to outpace them.

The paper seeks to close the scaling gap by jointly improving sequence‑wise information flow (via Kimi Delta Attention) and expert load balance (via Quantile Balancing), enabling a 2.8 T‑parameter multimodal MoE with a 1‑M token window.

The shift toward 2.8 T‑parameter multimodal MoE models marks a decisive step for open‑source AI.

Model Overview

Kimi K3’s architecture, scaling specs, and key components for trillion‑parameter multimodal MoE.

Kimi K3 expands the Kimi family to a trillion‑parameter scale while supporting a one‑million‑token context. Its design simultaneously enlarges model width, depth, and sequence capacity.

Kimi K2 is the previous‑generation multimodal MoE model with roughly one‑trillion total parameters and thirty‑two‑billion activated parameters.

The architecture adds Kimi Delta Attention (KDA) for efficient long‑sequence mixing, interleaved with Gated MLA layers that preserve global interaction. Attention Residuals let each layer attend to any prior layer, while Stable LatentMoE expands the expert pool to 896 experts with 16 active per token. SiTU‑GLU activation and Quantile Balancing keep optimization stable at extreme sparsity.

**Figure.** Scaling law curves for Kimi K2 and Kimi K3. Kimi K3 achieves 2.5× gain in scaling efficiency.

Post‑training augments the pre‑trained foundation with reinforcement learning across long‑horizon coding, general agents, reasoning, and knowledge tasks, each spanning multiple effort levels. Training environments range from verifiable search and professional knowledge work to multimodal vision‑in‑the‑loop tool use and autonomous execution, producing a unified model via multi‑teacher on‑policy distillation.

Infrastructure co‑design includes fused kernels, KDA Context Parallelism, and state‑aware prefix caching for efficient attention; MoonEP delivers perfectly balanced expert execution with static shapes and zero‑copy communication; and a co‑located RL system combines partial rollouts, external KV‑cache retention, adaptive throttling, and resumable microVM sandboxes to sustain million‑token trajectories.

2.8 T total parameters, 104 B activated parameters, and a 1 M‑token context window.

Core Architecture and Mechanisms

Three orthogonal mechanisms—KDA, SiTU‑GLU, and Quantile Balancing—enable scaling across sequence, depth, and width.

The architecture tackles three bottlenecks—long‑range token mixing, depth‑wise information loss, and channel‑wise expert overload—by inserting a single, repeatable trick in each dimension.

KDA adds a per‑channel forget gate to the delta‑rule recurrence, letting each hidden dimension decay at its own rate while still writing new content.

Compute $S_1 = \operatorname{Diag}(\alpha_1)S_0 + \beta_1 k_1 v_1^{\top}$ → $S_1$ becomes a $4\times4$ matrix with each column equal to $0.5\,k_1$ scaled by the corresponding $\beta$ entry.

For token 2, decay $S_1$ by $\operatorname{Diag}(\alpha_2)$ (e.g. $\alpha_2=(0.85,0.75,0.65,0.55)$) and add the new write term using its own $k_2$, $v_2$, $\beta_2$.

The resulting $S_2$ shows larger values in channels with higher $\alpha$ (slow decay) and smaller values where $\alpha$ is low, illustrating channel‑wise memory.

KDA lets each hidden channel retain information for a different number of tokens, so the model can simultaneously store short‑term details and long‑term context without blowing up memory.

How does KDA differ from standard linear‑attention decay?

Standard linear attention applies a single scalar decay to the whole vector, erasing all dimensions at the same rate. KDA uses a diagonal decay matrix, so each channel decays independently, and it couples decay with a gated low‑rank write, which together preserve useful long‑range signals while still allowing rapid adaptation.

SiTU‑GLU caps both the gate and the up‑branch with smooth hyperbolic‑tangent limits, preventing the unbounded growth that SwiGLU can produce.

Gate pre‑cap: $W_g x = 8$ → $\beta_1\tanh(8/4)=4\tanh(2)\approx3.93$.

Gate sigmoid: $\operatorname{Sigmoid}(8)\approx0.9997$.

Up‑branch pre‑cap: $W_u x = 8$ → $\beta_2\tanh(8/25)=25\tanh(0.32)\approx7.86$.

Final output: $3.93 \times 0.9997 \times 7.86 \approx 30.9$, well below the theoretical bound $4\times25=100$.

The tanh caps ensure that even large inputs cannot push the activation beyond a predictable ceiling, stabilizing training of the MoE.

Why not just use the standard GLU or SwiGLU?

GLU’s sigmoid gate is bounded but the up‑branch is linear, so large inputs still produce unbounded outputs. SwiGLU replaces the sigmoid with Swish, which is also unbounded, leading to occasional activation spikes. SiTU‑GLU caps both sides, eliminating those spikes while preserving the beneficial non‑linearity.

QB adjusts a per‑expert bias so that, after bias addition, exactly the target number of tokens selects each expert, achieving perfect load balance without an auxiliary loss.

Add current biases (zero) → margins equal $s$.

Target load $q = mk/n = 4\cdot1/2 = 2$ tokens per expert.

For expert 1, the $(1-k/n)=0.5$‑quantile of column 1 is $0.55$ (median of 0.8,0.6,0.3,0.5). Set $b_1^{(1)} = -0.55$.

For expert 2, median of column 2 is $0.45$, so $b_2^{(1)} = -0.45$.

Adjusted scores become: $\begin{bmatrix} 0.25 & -0.25\\ 0.05 & -0.05\\ -0.25 & 0.25\\ -0.05 & 0.05 \end{bmatrix}$. Top‑1 per row now selects expert 1 for rows 1,2 and expert 2 for rows 3,4, achieving the balanced load (2,2).

Because the bias is set to the median of each column, exactly half the tokens outrank it, guaranteeing the desired per‑expert count without any extra loss.

How does QB avoid the auxiliary‑loss tricks used in earlier MoE routers?

Earlier routers add a regularization term that penalizes load imbalance, which requires tuning a weight and incurs extra gradient computation. QB instead directly manipulates the routing scores via a bias derived from a quantile, so the routing objective stays pure (maximizing token‑expert affinity) and balance is achieved analytically in a single forward pass.

**Figure 2.** The Kimi K3 architecture, organized around token, channel, and layer mixing, with a native vision pathway at the input. Each block contains three Kimi Delta Attention (KDA) layers followed by one Gated MLA layer, with each attention layer paired with a Stable LatentMoE feed-forward network. Attention Residuals (AttnRes) use learned pseudo-queries ($w$) to derive attention weights ($\alpha$) over the embedding and preceding block outputs, enabling selective information flow across depth. **Top left:** the Stable LatentMoE module with shared and routed experts. **Bottom left:** the KDA module. **Bottom right:** the native vision pathway.

**Figure 3.** Lower-bounded decay and its effect on chunkwise KDA computation. (a) Kimi Linear uses an unbounded negative-Softplus mapping, whereas Kimi K3 bounds the log-decay with a scaled sigmoid; the curves show $A = 0$ and $g_{min} = -5$. (b) Kimi Linear evaluates each diagonal tile with an explicit position-pair computation, while the bounded range in Kimi K3 allows all causal tiles to use dense Tensor Core matrix multiplications.

**Figure 4.** Gate and up branches of GLU, SwiGLU, and SiTU-GLU, together with their scalar responses, where $\sigma$ denotes the sigmoid function. Both branches receive the scalar input $x$, and all curves share the domain $x \in [-10, 100]$; the inset magnifies the near-origin region. SiTU-GLU, shown in red with $\beta_1 = 4$ and $\beta_2 = 25$, closely follows SwiGLU near the origin and approaches the bound $|f(x)| \leq \beta_1\beta_2 = 100$ for large positive inputs, whereas SwiGLU remains unbounded.

**Figure 5.** Illustration of Quantile Balancing with $m = 8$ tokens, $n = 4$ routed experts, and $k = 1$ selected expert per token. (a) Token-wise Top-$k$ routing (tokens on the left, experts on the right) produces loads $(4, 3, 1, 0)$; darker circles indicate overheated experts, whereas faded and dashed circles indicate underutilized and dying experts, respectively. (b) Each gray bar is the margin of the currently biased score, $s_{i,j} + b_j^{(t)} - \alpha_i^{(t)}$, so the row-wise maxima reproduce the routing in (a). The dashed red line in each column is the bias adjustment $b_j^{(t)} - \widehat{b}_j^{(t+1)}$, placed at the $(q+1)$-th largest margin so that exactly $q = 2$ margins exceed it. The marker $\star$ denotes the row-wise Top-$k$ choice after subtracting the column adjustments, i.e., the routing in (c). (c) The retained choices yield the balanced load $(2, 2, 2, 2)$; red edges denote assignments changed by QB.

Pre-Training Data and Strategy

We detail the curated multimodal corpus, training recipe, and progressive context extension that enable Kimi K3.

Scaling multimodal data and million‑token contexts strains compute and memory budgets. A unified pre‑training pipeline that curates high‑quality text and vision data, then gradually expands context, is essential to keep training economical.

Scaling‑law curves on held‑out OOD validation reveal an approximately 2.5× gain in overall scaling efficiency over Kimi K2.

Measured across batch size, learning‑rate, tokens‑per‑parameter ratio, and model shape after incorporating the new data and curriculum.

Instead of grafting a vision encoder onto a pre‑trained language model, we train a single backbone on interleaved visual‑text tokens from day 1, so the model learns a unified representation without a costly alignment stage.

How does this differ from the common “pre‑train language, then align vision” workflow?

The common workflow freezes a language backbone and later learns a mapping from vision features to language space, which incurs a domain‑shift penalty and requires extra alignment data. Our strategy trains both modalities together, so the mapping emerges organically within the shared attention layers.

We let the model acclimate to longer sequences by slowly stretching the context window, much like a reader gradually tackles longer chapters to build stamina.

Step 1: Model sees tokens 1‑8 and learns short‑range dependencies.

Step 2: After 10 k steps, window expands to 16 tokens; the model now processes tokens 1‑16, re‑using the learned 8‑token patterns as a foundation.

Step 3: After another 20 k steps, window expands to 32 tokens; the model must now attend across the full 32‑token span, forcing it to integrate information from the earlier halves.

Throughout, the learning‑rate is briefly warmed up at each expansion to keep gradients stable.

This staged growth lets the model reuse previously learned short‑range structure while gradually acquiring long‑range reasoning, avoiding the instability of jumping directly to a 1 M‑token window.

Why not simply train with the full 1 M‑token window from the beginning?

Training from scratch on a million‑token context overwhelms memory and forces the attention layers to learn global patterns before any local structure exists, leading to divergence or severe under‑utilization of compute. The curriculum supplies a scaffold of short‑range context that the model can safely expand.

**Figure 6.** Vision-tower gradient norms in our pre-training ablations. Compared with the SigLIP-initialized MoonViT-3D, the from-scratch MoonViT-V2 maintains lower gradient norms with fewer spikes, indicating more stable optimization.

Curated multimodal corpora spanning text, code, math, and vision, combined with a progressive context curriculum, give Kimi K3 a stable foundation for unified multimodal learning.

Post-Training Pipeline

How Kimi K3 turns fine‑tuned experts into a single, deployment‑ready agent.

Scaling Kimi K3 to multimodal, long‑horizon agents requires more than a single fine‑tuning pass; latency spikes and token‑budget drift become fatal at scale.

The pipeline stitches three orthogonal stages—Supervised Fine‑Tuning, Reinforcement Learning, and Multi‑Teacher On‑Policy Distillation—so that each stage adds a distinct capability without re‑learning what previous stages already mastered.

Stage 1 (SFT) fine‑tunes $\theta$ on the 4 prompts, achieving 2‑token answers.

Stage 2 (RL) runs the partial‑rollout scheduler with K = 2 completions and $\lambda$ = 0.5, so training proceeds after the first completion finishes.

Stage 3 (MOPD) distills the three expert policies into a single model $\theta$′, preserving the 2‑token answer length while inheriting the high‑effort expert’s richer tool usage.

The three‑stage design lets the final model keep the SFT’s data efficiency, the RL’s domain expertise, and the MOPD’s unified deployment footprint.

How does this pipeline differ from simply fine‑tuning a single model on all tasks?

Fine‑tuning end‑to‑end forces the model to learn every capability from scratch, which quickly saturates the token budget and inflates latency. By isolating expertise in RL experts and then distilling, we keep the SFT’s data efficiency while adding high‑level reasoning only where needed.

SFT supplies a cold‑start policy that already knows how to serialize agentic trajectories, so RL never starts from a clueless baseline.

Why is the XTML chat template needed for SFT?

Agentic tasks involve nested tool calls and multi‑turn reasoning. XTML tags each turn and tool invocation, turning a heterogeneous dialogue into a flat token sequence that the transformer can attend to uniformly.

The RL stage trains nine domain‑expert policies while avoiding the “wait‑for‑the‑slowest” bottleneck that plagues long‑horizon rollouts.

Iteration 1: launch 4 rollouts; completions 1 and 2 finish after 3 seconds.

Policy update runs immediately on the finished rollouts.

Rollouts 3 and 4 are enqueued; iteration 2 resumes them while launching new rollouts for the next batch.

Partial rollout decouples rollout latency from update frequency, preventing a single long trajectory from stalling the entire training loop.

How does partial rollout avoid waiting for stragglers?

By committing to an update as soon as $\lambda$ · K rollouts finish, the scheduler discards the need for all trajectories to terminate before the next gradient step. The unfinished rollouts are simply deferred, keeping the training pipeline continuously fed.

Each problem receives a budget proportional to its estimated difficulty; scaling $\tau$ lets us sweep from generous to frugal reasoning effort.

High‑effort phase: effective budget = $\tau$ · b₀ = 18 tokens; the policy may generate up to 18 tokens before the reward is clipped.

Low‑effort phase: effective budget = 9.6 ≈ 9 tokens; the policy is penalized if it exceeds this limit.

During inference, the model selects the appropriate expert based on the desired $\tau$ value.

Adjusting $\tau$ directly trades off reasoning depth for token efficiency without changing the underlying model architecture.

What happens if $\tau$ is set too low?

The policy is forced to truncate its reasoning prematurely, often resulting in incomplete tool calls or missing intermediate verification steps, which degrades downstream performance on complex tasks.

GRM replaces a static scalar reward with a human‑readable rubric, letting the model judge candidates on multiple criteria while a verbosity cap prevents reward hacking.

How does the verbosity control stop reward hacking?

Any candidate that exceeds the scaled length threshold $\sigma$ · ℓ₀ is automatically assigned the losing side of the binary comparison, making it impossible to gain reward by simply adding filler tokens.

MOPD treats each of the nine RL experts as a teacher and gives the student a dense per‑token advantage signal that is clipped to stay stable.

Why use a stop‑gradient operator in the OPD reward?

Without stop‑gradient the teacher would receive gradient updates from the student’s loss, turning the distillation into a two‑way interaction that can corrupt the expert’s already‑trained policy.

Only the massive MoE expert weights are quantized to 4‑bit, keeping the rest of the model in higher precision to preserve accuracy while slashing memory.

Why quantize only the expert weights and not the rest of the model?

The experts dominate the parameter count; quantizing them yields the biggest memory savings. The remaining modules are already lightweight, and keeping them in higher precision avoids the accuracy loss that would arise from quantizing the attention projections and routers.

We train a lightweight draft model to predict the next token; the acceptance rate LLK measures how often the draft’s prediction matches the target model’s distribution.

Why optimize the LK loss instead of the usual KL divergence?

KL penalizes divergence even when the two distributions already agree on the most probable token, whereas LK directly measures the acceptance probability of the draft’s top‑k sample, which is the actual metric that determines speedup in speculative decoding.

The environment treats an agent harness as a plug‑in collection of modules (tools, prompts, memory, sub‑agents), allowing us to instantiate many different harness configurations on the fly.

Why expose the model to multiple harness configurations?

Different harnesses encode different prompting styles, tool APIs, and context‑management policies. Training across them forces the model to learn the underlying semantics of tool use rather than memorizing a single harness’s idiosyncrasies.

A hierarchical knowledge graph supplies fine‑grained concepts that drive web retrieval, ensuring both breadth (coverage) and depth (specialization) of synthesized tasks.

How does the graph avoid duplicating concepts?

When an agent proposes a new node, it first searches the existing graph for semantically equivalent or closely related concepts; if found, the proposal is merged rather than added, keeping the DAG sparse.

Each task is a multi‑step interaction with a sandboxed tool (e.g., a Python interpreter) that returns concrete observations, enabling automatic verification of every intermediate step.

We reward agents that produce GPU kernels matching expert implementations in both correctness and performance, while penalizing hacking tricks.

Mock versions of real applications (Gmail, Notion, Slack) let the model practice long‑horizon workflows without external API limits.

AETs present a goal, constraints, and a verifier; the agent must plan, act, and adapt without any demonstration trajectory.

Agents synthesize full‑stack web applications from textual specifications, then run them in a container to verify functional correctness.

**Figure 8.** Scores and the average assistant steps across a variety of public and in-house evaluations during RL. By scaling RL FLOPs, tool-call steps scale up consistently, accompanied by a comprehensive improvement in the model's overall capability.

**Figure 9.** Overview of knowledge-graph-guided task synthesis. The hierarchically organized knowledge graph represents concepts at multiple levels, ranging from broad domains to fine-grained concepts. Related nodes are sampled to form a keyword set that guides the retrieval of publicly available source materials. For each synthesis instance, the system selects a task type and uses the retrieved materials to synthesize a corresponding task.

System Infrastructure

Infrastructure scaling co‑designs attention, sparse training, and agentic workloads for efficient million‑token operation.

Scaling a multimodal Mixture‑of‑Experts model to million‑token contexts forces three intertwined problems: (1) the recurrent KDA state must be moved efficiently across devices, (2) expert loads become wildly imbalanced, and (3) long‑horizon RL requires reusable prefixes without exploding memory.

KCP treats each sequence segment as an independent mini‑recurrence, computes its local contribution, then stitches the pieces together with a prefix‑scan—exactly like merging sorted sub‑lists where each thread produces a locally sorted chunk and a final scan merges them.

Rank 0 computes $\tilde S^{[0]} = [1,0] + [1,0] + [1,0] + [1,0] = [4,0]$ and $M_{4\leftarrow1}^{[0]} = I$.

Rank 1 computes $\tilde S^{[1]} = [1,0] + [1,0] + [1,0] + [1,0] = [4,0]$ and $M_{4\leftarrow1}^{[1]} = I$.

All‑gather shares $M$ and $\tilde S$; prefix scan yields $S^{[1]} = \tilde S^{[1]} + M^{[1]} \cdot S^{[0]} = [4,0] + I\cdot[4,0] = [8,0]$.

The final state after token 8 is $[8,0]$, exactly the sum of all local writes.

KCP shows that even though each chunk sees only its own tokens, the global recurrent state is recovered exactly by a constant‑size all‑gather and a cheap prefix scan.

How does KCP differ from naïve context parallelism that simply sums per‑chunk KV caches?

Naïve summation ignores the token‑dependent transition $M_t$, so the resulting state would be $\tilde S$ only. KCP adds the $M_{t\leftarrow1}$ term, which propagates the incoming state through each chunk’s dynamics, preserving the exact recurrence.

FlashKDA overlaps the intra‑chunk attention work with the inter‑chunk state propagation, turning what would be idle GPU cycles into useful computation.

Balanced expert execution is the next bottleneck: without perfect load distribution, some ranks sit idle while others stall the pipeline.

MoonEP3 guarantees that every EP rank processes exactly $S\!K$ tokens per micro‑batch, eliminating the idle‑time spikes that plague conventional routers.

Activations are treated as pluggable resources: each tensor declares a storage policy (FP8 quant, offload, recompute) that the manager enforces without touching model code.

Prefix KV caches are written back to CPU DRAM only when evicted, allowing idle prefixes to sit cheaply while active ones stay on GPU.

AgentENV isolates each agentic execution in a Firecracker microVM, providing strong security guarantees without the latency of full containers.

Both the fixed‑size KDA state and the variable‑size MLA KV cache are stored in a unified paged pool, enabling joint reuse at any 512‑token hash boundary.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers