Motif 3: Technical Report

Junghwan Lim, Joon Son Chung, Sungmin Lee, Wai Ting Cheung, Gihun Cho, Minsu Ha, Sangho Kang, Beomgyu Kim, Dongseok Kim, Jangwoong Kim, Taehyun Kim, Taewhan Kim, Jeesoo Lee, Jeongdoo Lee, Junhyeok Lee, Dongpin Oh, Hyeyeon Cho, Dahye Choi, Jaeheui Her, Hanbin Jung, Changjin Kang, Minjae Kim, Youngrok Kim, Hyukjin Kweon, Hongjoo Lee, Yeongjae Park, Bokki Ryu

Motif 3 is a 314B-parameter Mixture-of-Experts model using Grouped Differential Latent Attention for efficient long-context reasoning.

How does Motif 3 achieve high-performance language modeling at scale using a fine-grained Mixture-of-Experts architecture and specialized training optimizations?

Large language models struggle to balance high expert capacity with the computational overhead of processing every token through dense layers. Motif 3 uses a fine-grained Mixture-of-Experts architecture that routes each token to only eight of 384 available experts, paired with Grouped Differential Latent Attention to compress key-value states while maintaining selective attention. This design enables a 314-billion-parameter model to activate only 13.2 billion parameters per token, achieving competitive performance on agentic and reasoning tasks.

Paper Primer

The core innovation is Grouped Differential Latent Attention (GDLA), which merges differential attention's noise-suppression mechanism with the memory-efficient latent key-value representations of Multi-head Latent Attention. This allows the model to focus on relevant context while drastically reducing the memory footprint of the key-value cache during inference.

To stabilize training at this scale, the authors employ Expert-Specific PolyNorm activations, which learn independent polynomial responses for each expert, and modified manifold-constrained hyper-connections that prevent activation outliers by annealing residual scaling factors.

Motif 3 achieves superior training efficiency compared to standard attention architectures.

In controlled 10B-parameter experiments, GDLA reached a loss of 3.2 using 9.2% fewer training tokens than Multi-head Latent Attention. 9.2% reduction in training tokens for equivalent loss.

The model maintains high performance on long-horizon tasks despite its sparse activation.

Evaluation across agentic, mathematical, and hallucination-sensitive benchmarks shows competitive results against leading open-weight models. 13.2B parameters activated per token out of 314B total.

Why use a Mixture-of-Experts (MoE) design with such a high number of experts (384)?

This fine-grained sparsity provides a large pool of expert capacity, allowing the model to specialize across diverse domains like STEM, code, and multilingual content without increasing the computational cost per token.

How does the model handle the memory demands of 256K-token context lengths?

The system uses a hybrid context-parallel strategy: it applies window-aware Ring Attention for sliding-window layers and Ulysses all-to-all communication for full-attention layers, combined with full activation recomputation to minimize memory footprint.

Motif 3 demonstrates that combining fine-grained expert routing with compressed latent attention allows for massive parameter counts that remain computationally tractable for long-context agentic applications.

Introduction and Overview

We frame the scaling bottlenecks of large MoE models and motivate Motif 3’s design.

Current Mixture‑of‑Experts (MoE) models struggle to scale because routing many experts per token creates heavy communication overhead, and naive expert selection often leads to overload or starvation of experts during training.

To unlock the full potential of large MoE models we must route a small, diverse set of experts per token while keeping inter‑device communication cheap and ensuring all experts stay useful throughout training.

Motif 3 demonstrates that fine‑grained expert routing combined with hierarchical training can scale MoE models to hundreds of billions of parameters without prohibitive communication costs.

Model Architecture

This section details the GDLA token‑mixing layer, fine‑grained MoE routing, and supporting mechanisms.

Standard self‑attention spreads attention mass over many irrelevant tokens, inflating compute and memory. Motif 3 addresses this waste by introducing a token‑mixing layer that suppresses noise while expanding signal capacity, and by routing only a handful of experts per token.

GDLA treats attention like a two‑mic setup: a “signal” mic captures the useful context, while a “noise” mic records redundant patterns; the model then subtracts the noise mic (scaled per token) from the signal mic, leaving a cleaner representation.

Repeat the noise output twice to match the four signal heads: $\text{Repeat}_2(H_{N,t}) = [0.2, 0.1, 0.2, 0.1]$.

Scale the repeated noise by $\lambda_t$: $\lambda_t \odot \text{Repeat}_2(H_{N,t}) = [0.18, 0.08, 0.17, 0.075]$.

Subtract from the signal heads: $D_t = H_{S,t} - (\lambda_t \odot \text{Repeat}_2(H_{N,t})) = [0.62, 0.52, 0.53, 0.425]$.

Apply a sigmoid gate $G_t = [0.6, 0.7, 0.5, 0.8]$ element‑wise: $\sigma(G_t) \odot D_t = [0.37, 0.36, 0.27, 0.34]$.

Flatten and project with $W_O$ (e.g., identity for illustration) to obtain the final output vector.

Repeating the noise output lets each signal head benefit from the same noise estimate, while the per‑head coefficient $\lambda_t$ lets the model attenuate noise differently for each token.

How does GDLA differ from the earlier Grouped Differential Attention (GDA) formulation?

GDA uses the same number of heads for signal and noise, allocating equal capacity to both. GDLA introduces a grouped ratio $g$, giving $g$ times more signal heads than noise heads and sharing the noise heads across signal groups. This asymmetry boosts signal modeling power while keeping the extra noise computation modest.

The GDLA schedule interleaves one full causal‑attention layer with three sliding‑window layers, all using the same GDLA formulation. Queries and keys are first projected to low‑rank latents $c^Q_t$ and $c^{KV}_t$, normalized by RMSNorm, and then expanded into 16 KV heads shared by both signal and noise paths. An element‑wise gate derived from $c^Q_t$ modulates the differential output before the final linear projection.

Instead of activating a large block of experts for every token, the model selects a tiny, token‑specific subset of eight experts from a pool of 384, dramatically reducing per‑token compute while still exposing a huge overall capacity.

Why route only eight experts per token instead of a larger number?

Eight experts strike a balance: they provide enough diversity to capture varied contexts while keeping the compute per token modest. Adding more experts would increase FLOPs roughly linearly, eroding the efficiency gains of sparsity.

Motif 3 replaces the classic residual addition with modified manifold‑constrained hyper‑connections (mHC). mHC expands a single residual stream into $n=4$ parallel streams, learns token‑dependent reduction and redistribution matrices, and scales the post‑mapping factor $s_t$ from 2 to 1 during pretraining to avoid activation blow‑up. Expert‑Specific PolyNorm replaces the shared SiLU gate with a per‑expert polynomial activation, learning separate coefficients $a_{i,n}$ and bias $b_i$ for each expert. Finally, a multi‑token prediction (MTP) head is added as an auxiliary pretraining objective, enabling self‑speculative decoding without altering the main autoregressive path.

MLA compresses the key‑value cache by projecting the full KV matrix into a low‑rank latent space, then expands it back only when needed, cutting memory usage during inference.

In what way does MLA reduce KV‑cache requirements compared to standard multi‑head attention?

Standard attention stores a separate key and value vector for each head, scaling linearly with the number of heads. MLA stores a single low‑rank latent representation for all heads and reconstructs the full KV only inside the attention computation, cutting the stored KV size by the rank reduction factor.

**Figure 1.** Illustration of the GDLA (Grouped Differential Latent Attention) architecture of Motif 3. **Left:** each Transformer block pairs a GDLA token-mixing layer with a sparse MoE channel-mixing layer and uses mHC to mix $n$ parallel residual streams. **Bottom right:** the query and KV paths use low-rank projections. The complete KV latent $\mathbf{c}^{KV}$ is RMS-normalized and expanded once by $\mathbf{W}^{KV}_b$ into 16 KV heads shared by the signal and noise query paths. The decoupled rotary key is also shared by all KV heads. An element-wise gate computed from the normalized query latent $\mathbf{c}^Q$ modulates the differential output before $\mathbf{W}^O$. **Top right:** $n_S = g n_N$ signal heads are paired with repeated noise-head outputs. A token-dependent coefficient $\lambda_t = \sigma(\mathbf{x}_t \mathbf{W}^\lambda) \in (0, 1)^{n_S}$ is predicted for every signal head and scales the noise term before subtraction.

**Figure 2.** Attention training-loss comparison. GDLA achieves lower loss than GDA and MLA and reaches a loss of 3.2 with 9.2% fewer training tokens than MLA.

**Table 1.** Summary of the principal architectural configuration and model dimensions used in Motif 3, including its hybrid attention pattern, fine-grained expert structure, and long-context support.

Distributed Training System

We detail the hierarchical parallel training system that stitches expert, data, and context parallelism into a low‑memory pipeline.

Training Motif 3 at extreme scale demands a system that can keep memory under control while fully utilizing the hardware fabric. The core trick is a hierarchical parallel layout that aligns expert, data, and context parallelism with the physical network topology, eliminating redundant collectives and overlapping communication with computation.

Arrange the three parallel dimensions—expert (EP), data‑shard (DP‑shard), and context (CP)—as nested groups that match the node‑level NVLink mesh, so each group can communicate locally without crossing the slower inter‑node links.

Step 1: EP groups dispatch their $4$ tokens to the local expert; each node sends $4$ tokens to its partner node via NVLink.

Step 2: DP‑shard groups shard the model parameters; each node holds half the parameters and performs forward/backward on its local tokens.

Step 3: CP reuses the EP process dimension; the two nodes exchange the trailing $W = 2$ tokens of the preceding shard (halo exchange) instead of the full $L = 8$ tokens.

Step 4: Overlapped Reduce‑Scatter reduces gradients from the two DP‑shard groups while the backward pass of the experts proceeds.

Step 5: The final All‑Gather for expert weights occurs only once after the last microbatch, reusing the resident MXFP8 copy.

This toy illustrates how nesting the parallel dimensions confines the bulk of communication to the fast intra‑node links and replaces a full $L$‑token exchange with a tiny halo of size $W$, yielding a $L/(2W)=2$× reduction in per‑rank traffic.

How does this hierarchical layout differ from a naïve flat data‑parallel scheme?

In a flat scheme every rank would need to All‑Gather full expert weights and exchange full $L$‑token KV shards, incurring $O(L)$ communication per rank. The hierarchy restricts the heavy All‑Gather to the EP dimension (NVLink) and replaces the $L$‑wide exchange with a $W$‑wide halo, cutting the per‑rank bandwidth by roughly $L/(2W)$.

Form EP groups of $8$ GPUs per node and enable HybridEP for expert dispatch/combine.

Instantiate a sharded data‑parallel (DP‑shard) group of $8$ ranks across nodes.

Assign the remaining ranks to the DP‑replicate dimension for replicated parameters.

Apply FSDP optimizations: overlap Reduce‑Scatter, suppress DP‑replicate sync on intermediate microbatches, and drop expert‑weight All‑Gathers.

Enable context parallelism (CP = $8$) by reusing the EP process dimension for long‑context sequences.

Run the model without pipeline parallelism, fitting the $314\,\text{B}$‑parameter model in GPU memory.

Overlapped gradient Reduce‑Scatter with a bounded in‑flight window.

Activation recomputation wraps each transformer block in a checkpoint region, re‑executing the forward pass during backward according to a per‑operator policy, which dramatically cuts peak activation memory.

We fuse the output projection and cross‑entropy into a single Liger kernel, processing hidden states in chunks so the full logits tensor never materializes, keeping loss scaling invariant to batch partitioning.

Low‑precision training stores expert tensors in MXFP8, using DeepGEMM1 as the grouped GEMM backend; activations are quantized before dispatch, halving communication cost and reducing quantization overhead by a factor of $8$ (top‑k).

**Figure 3.** Overall low-precision training recipe. Color encodes numerical precision: teal for MXFP8, blue for BF16, and peach for FP32. Dashed borders mark collectives that cross ranks. Only the row-wise MXFP8 weight is All-Gathered; the column-wise copy required by Dgrad is produced locally by a fused row-to-column transcode kernel. Expert activations are quantized once before EP dispatch, allowing the dispatch itself to use MXFP8. Gradient synchronization exchanges BF16 shards while performing each reduction locally in FP32.

**Figure 4.** Selecting the context-parallel algorithm per attention layer. (a) A packed sequence sampled from the training data and containing 473 documents, shown on the attention score matrix $QK^T$. (b) Rank-average forward-pass communication volume per layer. The marker above the Ring Attention bar reports the final rank, which receives twice the rank average. (c) Rank-average bytes sent per attention FLOP versus the window size $W$, evaluated on a single document. All panels use $P = 8$, $L = 256$K tokens, the head configuration of Section 4.3, and BF16. Panels (a) and (b) use $W = 128$ tokens.

**Figure 5.** Ring Attention communication under full and sliding-window attention. The block structure of $QK^T$ for $P = 8$ context-parallel ranks, shaded from the perspective of rank 5: query shards on the left, key/value shards on top, score blocks in between. Only the key and value shards are exchanged. Full attention requires every preceding shard, one per rotation step; a window of $W \leq L/P$ tokens requires only the trailing $W$ tokens of the preceding shard, reducing the per-rank volume from $\Theta(L)$ to $\Theta(W)$. Exchanged tokens are drawn far wider than $W/L$ to remain visible; forwarding hops of the standard ring schedule are omitted.

Table 3 lists the expert‑health metrics (dispatch stats, token counts, RMS, cosine similarity, etc.) that the training system monitors to detect starvation, traffic concentration, and numerical instability.

Pre-training and Base Performance

Motif 3’s pretrained base model sets new benchmarks across knowledge and code tasks.

Motif 3 achieves 86.20% accuracy on the MMLU 5‑shot benchmark.

Table 4 reports 86.20 for MMLU 5‑shot.

The provided image contains a table comparing various tokenizers across different metrics (Vocab, en, fr, ko, ja, zh, code, math).

**Figure 6.** MoE component training comparisons from controlled experiments using models with approximately 10 billion parameters. (a) Expert-Specific PolyNorm maintains a higher mean effective rank in the expert gate weights than SwiGLU across layers, indicating a more evenly distributed singular-value spectrum. (b) Decaying router noise reduces the maximum number of tokens assigned to an expert more rapidly and guides the routing distribution toward the median-load regime early in training.

**Table 4.** Evaluation results for the Motif 3 pretrained base model. CoT denotes chain-of-thought prompting.

Post-training and Distillation

Post‑training consolidates specialist teachers into a single, broadly capable model.

Recall that Motif 3 scales sparse MoE models by pairing fine‑grained expert routing with a hierarchical training system that minimizes communication bottlenecks.

MOPD lets a single student listen to several domain‑expert teachers, but the student only records the word each teacher actually says for the current token – like a note‑taker who writes down the spoken word from the relevant specialist while ignoring the rest of the conversation.

How does MOPD differ from conventional knowledge‑distillation that copies the full softmax distribution?

Standard distillation forces the student to match every token‑level probability vector from a single teacher, which is expensive and mixes unrelated domains. MOPD instead selects the single specialist relevant to the current example and uses only its scalar log‑probability for that token, dramatically reducing computation and preserving the distinct expertise of each teacher.

Motif 3 attains the highest reported score on the agentic $\tau$3‑Banking benchmark (35.3), surpassing all other models listed in Table 6.

Table 6 shows $\tau$3‑Banking = 35.3 for Motif 3, while the next‑best model records 30.1.

The table lists various "Teacher" categories and their corresponding "Coverage" descriptions.

**Figure 7.** Mean reward over cumulative RL compute for the six GRPO-trained specialist teachers. The lighter curves show the per-update reward measurements, and the darker curves show their smoothed trends.

**Table 6.** Evaluation results for Motif 3. An asterisk (*) indicates that the corresponding result for Motif 3 is evaluated on the public subset only.

Limitations and Conclusion

We outline the remaining constraints of Motif 3 and sketch avenues for extending its capabilities.

Limitations of the current model are threefold. First, the training and evaluation pipelines do not span the full diversity of real‑world tasks, domains, languages, interaction patterns, and deployment conditions, so performance may degrade on under‑represented or unseen tasks. Second, Motif 3 remains a pure‑text architecture, which precludes direct handling of visual inputs. Third, while the model supports long contexts, many long‑horizon scenarios demand more reliable state tracking, planning, and recovery than the evaluated trajectories provide.

Future work will address these gaps along four directions. We will explore novel architectures that cut training and inference costs further and scale beyond the 314‑billion‑parameter Motif 3. We also plan to push native context lengths past one million tokens while keeping both compute and memory footprints modest. Adding native visual processing for images and video will open multimodal applications, and we will reinforce long‑horizon agent abilities by enriching environments, extending interaction trajectories, and improving planning and memory mechanisms.

The appendix also records the two regexes used by the tokenizer. Stage 1 implements a conventional subword pattern that isolates word‑like units, common contractions, short digit runs, symbols, line breaks, and generic whitespace. Stage 2 builds on Stage 1 by allowing a repeated “space + letter‑run” group, enabling a single token to span multiple whitespace‑delimited words while preserving the original handling of digits, punctuation, and line breaks.

Finally, we provide the per‑rank communication‑volume formulas that underlie Figure 4. For Ring Attention with causal early exit, rank $r$ receives only the $r$ preceding key/value shards, yielding an average volume of $\frac{(P-1)}{2P}\,L\,(h\,k\,dqk + h\,v\,dv)$ elements per rank. By contrast, Ulysses’ all‑to‑all communication costs $\frac{(P-1)}{P^{2}}\,L\,\big[(h\,q + h\,k)dqk + (h\,v + h\,o)dv\big]$ per rank. Assuming $h\,o = h\,q$ and $h\,v = h\,k$, the average‑rank ratio simplifies to $2/3$, meaning Ring moves 33 % less data on average, while the busiest rank sees a $4/3$ ratio, reversing the ordering.

Motif 3’s current constraints—limited task diversity, text‑only modality, and insufficient long‑horizon support—guide a roadmap that includes cheaper scalable architectures, million‑token contexts, multimodal extensions, and stronger agentic planning.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers