MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement
Xiaomi LLM-Core Team, Zongming Qiao, Ziyue Hua, Zirui Ou, Zihao Yue, Zihan Jiang, Zhuo Huang, Zhiyang Chen, Zhixian Zheng, Zhipeng Xu, Zhengrui Ma, Yuyang Hu, Yuhang Dong, Yuechen Zhang, Yudong Wang, Yuanxin Liu, Yixin Yang, Yishuo Cai, Yikai Zhao, Yihan Yan, Yifan Zhang, Yifan Song, Xiyu Wei, Xing Zhang, Xin Zhang, Xiaoqian Liu, Xiaodong Ji, Xiangwei Deng, Xueyu Guo, Wenhan Ma, Weimin Xiong, Weikun Wang, Weiji Zhuang, Shuo Liu, Shuhuai Ren, Shuhao Gu, Shimao Chen, Shijie Cao, Shihua Yu, Shicheng Li, Shengjie Zhou, Shaolei Zhang, Rang Li, Qiying Wang, Qingkai Fang, Qianli Chen, Minzheng Wang, Liwen Wang, Linli Yao, Linghao Zhang, Liangyu Cheng, Liang Zhao, Lei Li, Jinhao Dong, Jinyu Xiang, Jianyu Wei, Jiangshan Duo, Huaqiu Liu, Huanjie Fan, Hongyi Guan
MiMo-V2.6 scales reinforcement learning for agentic self-improvement via asynchronous compute, groupwise grading, and robust environment synthesis.
How can we scale reinforcement learning (RL) for agentic foundation models to achieve reliable self-improvement across diverse, multi-modal environments?
Foundation models struggle to achieve recursive self-improvement because current reinforcement learning (RL) pipelines lack the compute scale, diverse environments, and fine-grained reward signals necessary to guide complex agentic behavior. The authors introduce the MiMo-V2.6 series, which scales RL by decoupling control and data planes for asynchronous training, implementing groupwise agentic grading to distinguish solution quality, and using multi-harness training to ensure cross-task generalization. On the DeepSWE coding benchmark, this approach improves the MiMo-V2.6-Pro model from 58.4 to 72.6 in average@3 score, demonstrating consistent gains across coding, general workflows, and cybersecurity tasks.
Paper Primer
The core mechanism hinges on scaling RL compute across three dimensions: massive asynchronous rollout, diverse environment synthesis, and groupwise agentic grading. The system acts like a high-throughput factory: it runs thousands of parallel agent simulations, uses an agentic grader to rank the quality of the outputs within each batch, and redistributes reward signals to prioritize efficient, high-quality trajectories over merely "passing" ones.
Scaling RL compute significantly boosts complex problem-solving capabilities across diverse domains.
Performance on the DeepSWE coding benchmark improved from 58.4 to 72.6 for the 1.02T-parameter MiMo-V2.6-Pro model. A 24% relative improvement in coding task success rate over the course of RL training.
Groupwise grading provides more informative learning signals than binary test outcomes.
By comparing multiple solutions within a rollout group, the system distinguishes between high-quality and low-quality passing solutions, guiding the model toward shorter, more token-efficient implementations. The system maintains a confirmed reward-hacking rate of below 2% throughout training.
Why is "groupwise" grading necessary instead of just using standard test-case pass/fail rewards?
Binary test rewards are often too coarse; they cannot distinguish between a clean, efficient solution and a bloated, "hacky" one that happens to pass. Groupwise grading allows the model to learn from the relative quality of successful trajectories, steering it toward more reliable and efficient problem-solving behaviors.
What is the primary engineering challenge in scaling RL to this level of compute?
The main hurdle is maintaining training stability and throughput while processing massive, long-horizon trajectories. The authors address this by decoupling the control and data planes, using partial rollouts to keep GPUs saturated, and implementing rigorous adversarial screening to prevent reward hacking.
This paper establishes that scaling RL compute—when paired with agentic grading and robust environment isolation—is a viable path toward more capable, self-improving foundation models. Researchers can now use the open-sourced MiMo-V2.6 framework to study agentic RL without building the entire infrastructure from scratch.
Introduction to MiMo-V2.6
Why scaling agentic RL demands more than static supervised fine‑tuning.
Scaling agentic reinforcement learning (RL) cannot rely on static Supervised Fine‑Tuning (SFT) alone. To move beyond SFT we must build diverse, verifiable environments and a robust infrastructure that can sustain the massive compute and stability demands of large‑scale rollout collection.
Scaling agentic RL means replacing a fixed, pre‑trained policy (SFT) with a dynamic loop that repeatedly gathers trajectories, evaluates them in rich environments, and updates the model using high‑quality reward signals.
The paper highlights two major obstacles: first, foundation models need architectures that can absorb long‑horizon interactions and a sufficiently rich exploration space; second, scaling RL demands engineering solutions for high‑throughput rollout collection, consistent grading, and stable training pipelines.
The essential shift is from static SFT to large‑scale agentic RL that continuously learns from diverse, verifiable tasks.
Scaling RL Compute
Scaling RL compute by parallel rollout, partial execution, and dynamic sampling.
Scaling RL compute is the bottleneck that stalls higher‑capability models: rollout, grading, and training together demand thousands of GPUs and careful batching to avoid idle hardware.
Step 1: Prompt 1 produces candidates $o_{1,1}$ and $o_{1,2}$; Prompt 2 produces $o_{2,1}$ and $o_{2,2}$.
Step 2 (partial rollout): after $5$ tokens each, we pause all four sequences, rebuild their KV caches, and resume in the next rollout phase.
Step 3: The grader evaluates the four candidates, marking $o_{1,1}$ and $o_{2,2}$ as “all‑pass” (discarded) and keeping $o_{1,2}$, $o_{2,1}$.
Step 4: The dynamic sampler drops the discarded groups, leaving $2$ effective sequences whose advantages $A$ are $+0.7$ and $-0.3$.
Step 5: Training aggregates the retained tokens, computes the loss $L(\theta)$ using the masked advantage terms, and updates $\theta$.
Partial rollout hides the expensive KV‑cache rebuild behind the large batch, while the dynamic sampler guarantees that every retained token carries a non‑zero advantage signal.
How does this differ from simply increasing the batch size in standard RL?
Plainly enlarging the batch would still require each sequence to finish a full rollout before the next policy update, incurring a large KV‑cache re‑prefill cost each time. The presented method interleaves rollout and training: partial rollout pauses unfinished sequences, re‑prefills their caches once per policy change, and the dynamic sampler removes groups that would contribute no learning signal, so compute grows without proportional waste.
Partial rollout with dynamic sampling – keeps the large batch saturated while discarding useless groups.
With rollout consuming 43.8 % of the compute budget, training 43.5 %, and grading the remaining 12.7 %, the three‑part design keeps all GPUs busy and stabilizes gradients across billions of tokens.
Model Architecture
MiMo‑V2.6 interleaves local and global attention with multi‑token prediction to scale multimodal processing.
Scaling multimodal reinforcement learning demands an attention backbone that stays cheap while still seeing the whole context. A naïve Transformer would explode in memory as visual and audio streams grow, making large‑scale rollout collection infeasible.
The model alternates a “local zoom lens” (Sliding Window Attention) with a “global wide‑angle lens” (Global Attention) and then predicts several future tokens in parallel, keeping compute linear in sequence length while still gathering full‑sequence context.
Step 1 – SWA 1 on tokens $[t_0\!-\!t_3]$ produces local context vectors $c_0\!-\!c_3$.
Step 2 – GA 1 attends all eight tokens, mixing $c_0\!-\!c_3$ with $t_4\!-\!t_7$ to yield global vectors $g_0\!-\!g_7$.
Step 3 – SWA 2 processes windows $[t_0\!-\!t_3]$ and $[t_4\!-\!t_7]$ again, now using the updated globals $g_i$ as inputs.
Step 4 – GA 2 aggregates the refreshed tokens, producing the final backbone hidden states $h_0\!-\!h_7$.
Step 5 – The MTP module takes $h_7$ (the anchor) and predicts the next $K=3$ tokens $[t_8, t_9, t_{10}]$ in parallel.
The alternating local‑global pattern lets each token see its immediate neighbors many times before a cheap global sweep, achieving full‑sequence awareness with only $O(L\cdot W)$ work.
How does MiMo‑V2.6 differ from a vanilla Transformer that simply stacks self‑attention layers?
Vanilla Transformers apply full self‑attention at every layer, incurring $O(L^2)$ cost. MiMo‑V2.6 replaces most layers with $W$‑sized Sliding Window Attention (linear in $L$) and inserts occasional Global Attention layers to refresh long‑range information. The sparse MoE FFN further limits per‑token computation, and the MTP head predicts several future tokens at once, removing the sequential sampling bottleneck.
**Figure 2** Overall architecture of MiMo-V2.6. Audio, visual, and text inputs are mapped into a shared token sequence and processed by the MiMo Hybrid-SWA backbone, followed by the language-modeling head and multi-token prediction (MTP) blocks.
Table 1 lists the concrete configurations for the Flash and Pro variants, highlighting the dramatic parameter scaling from 310 B to 1.02 T while keeping the active expert count modest.
Pre-Training Setup
Pre‑training builds omni‑modal foundations and switches to a matrix‑aware optimizer for large‑batch RL.
The first stage trains the language backbone on text‑only data, then the second stage fuses a pre‑trained vision and audio encoder to learn from a 32K‑to‑256K token omni‑modal corpus.
Muown updates hidden weight matrices by orthogonalizing the gradient and then rescales each row to keep its norm stable, preserving data efficiency when batch sizes grow.
How does Muown differ from standard AdamW?
AdamW adapts each parameter independently, while Muown treats an entire hidden matrix as a unit: it first removes components parallel to the current matrix (orthogonalization) and then enforces a uniform row norm, which preserves the matrix’s geometry and scales better to very large batches.
Orthogonalize: project $G$ onto the space orthogonal to $W$, yielding $G_{\perp}=G - \frac{W^\top G}{\|W\|_F^2}W \approx \begin{bmatrix}0.45 & -0.05\\-0.15 & 0.25\end{bmatrix}$.
Update $W \leftarrow W - \eta\,G_{\perp}$ with learning rate $\eta=0.1$, giving $W'=\begin{bmatrix}0.955 & 2.005\\3.015 & 3.975\end{bmatrix}$.
Row‑norm control: compute row norms $\|r_1\|=2.15$, $\|r_2\|=4.78$; rescale each row to target norm $2.5$, producing $W''=\begin{bmatrix}0.88 & 1.84\\1.57 & 2.93\end{bmatrix}$.
Orthogonalization removes gradient components that would merely scale the existing matrix, while row‑norm control keeps each row’s magnitude in check, preventing runaway growth as batch size increases.
**Table 4.** Weighted SFT data composition. Token counts are in billions. Values are rounded to one decimal place; totals use unrounded aggregate counts.
Environments and Grading
Scaling RL demands reproducible, diverse environments with trustworthy rewards.
Scaling agentic RL hinges on environments that are both realistic and fully auditable; without reproducible execution and trustworthy rewards, agents quickly learn to exploit loopholes rather than solve the intended tasks.
We turn raw artifacts—code patches, workflow files, or visual assets—into tasks whose correctness can be checked automatically, like turning a pull‑request description into a test suite that any implementation must pass.
Agent writes
Both test cases pass, so the reward is 1.
Auditor inspects the specification and confirms the tests cover exactly the described behavior.
Since no extra checks exist, the task is accepted as verifiable.
Even a minimal specification can be made verifiable by aligning test cases directly with the stated requirement, eliminating reliance on hidden implementation details.
How does Verifiable Task Synthesis differ from simply writing a unit test for any piece of code?
In ordinary unit‑test writing, the test is often crafted after the implementation and may encode incidental details. Here the specification is created first, and the test suite is derived automatically to match it, ensuring that passing the tests truly reflects the intended behavior.
Instead of rewarding each rollout independently, we look at a whole batch, zero‑out rewards for confirmed hacks, and shift the remaining advantage toward the highest‑quality passing solutions.
Set the hack reward to 0 → rewards become [1, 0, 0, 1].
Compute group mean = 0.5 and advantages A = [0.5, -0.5, -0.5, 0.5].
Quality factors f = [0.9, –, –, 1.0] for the two passing rollouts.
$\lambda$ = (0.5+0.5) / (0.9*0.5 + 1.0*0.5) = 1.05.
Redistribute: A' = [0.9*0.5*1.05, 0, 0, 1.0*0.5*1.05] ≈ [0.47, 0, 0, 0.53].
Redistribution preserves total advantage while rewarding the higher‑quality solution more, and the hack is completely excluded.
Why not simply increase the binary test reward for better solutions instead of performing groupwise redistribution?
Binary tests cannot distinguish quality beyond pass/fail; redistribution uses additional rubric scores and hack filtering to create a graded signal that reflects nuanced differences among passing solutions.
We treat the environment as a sealed sandbox, strip any artefacts that could leak the solution, and run a dedicated hack‑agent to probe for hidden shortcuts before training.
Agent lists
Agent decompiles the binary to recover the fix.
Agent applies the recovered fix, passes all tests, and receives full reward.
Hack‑agent later scans the environment, finds the leftover binary, and flags the leak.
Cleanup removes the entire
Small artefacts like stray binaries can constitute a complete solution leak; systematic cleanup eliminates this hidden channel.
Why isn’t it enough to rely on static test suites to prevent reward hacking?
Static tests only verify observable behavior; agents can still obtain the correct answer through hidden artefacts or external resources that the tests never see. Dynamic probing and environment hardening are required to close those covert channels.
Start from a minimal agent loop containing system prompt, tool definitions, and context manager.
Instantiate a mini‑harness for a target domain (Code, General, Visual, Cyber) by adding domain‑specific tools.
Run a batch of tasks drawn from the corresponding environment pool.
Collect rollouts, apply groupwise grading (GRS or GAR) and reward‑hacking mitigation.
Update the policy using the redistributed advantages.
Repeat, optionally swapping to a different mini‑harness to encourage cross‑harness generalization.
**Figure 4** The overall code agent task scaling pipeline. We build tasks from diverse sources and go through rigorous assessment stages to ensure the supervision is accurate and robust.
**Figure 5** The overall general-agent environment and verifiable-task synthesis pipeline. We first construct realistic, resettable environments from real-world files and software mocks. Agents then explore these environments and synthesize tasks with verifiable rubrics.
**Figure 6.** Reward hacking prevention and monitoring. (a) Environment preparation and iterative hack-agent screening before training, while offline trajectory audits monitor hacking behaviors throughout training. Findings from both stages are used for environment improvements. (b) Top: the fraction of environments found hackable over cleanup rounds. Bottom: the fraction of trajectories with detected reward hacking throughout our final RL run.
**Figure 7** Groupwise agentic grading for code-agent RL. (a) Groupwise reward synthesis combines test outcomes with per-rollout scores from precomputed task-specific rubrics. (b) Groupwise advantage redistribution compares trajectories online, resets confirmed-hack rewards to zero, and redistributes sequence-level advantages. Bars schematically illustrate the redistribution of positive advantage from lower- to higher-quality passing solutions.
Experimental Results
Scaling agentic RL needs diverse verifiable environments and robust infrastructure beyond SFT.
MiMo‑V2.6‑Pro reaches 72.57 % on DeepSWE v1.1, outpacing the Flash variant by 6.89 points.
Figure 3 (left panel) shows the final scores after 30 RL steps.
**Figure 3** Left: average@3 score on DeepSWE v1.1 versus cumulative RL cost. Right: training, rollout, and grader cost shares for MiMo-V2.6-Pro and MiMo-V2.6-Flash.
RL Infrastructure
We detail the RL infrastructure that powers massive, stable agentic rollouts via a penalty module and scalable data pipelines.
Large‑scale agentic RL demands an infrastructure that can keep thousands of rollouts flowing while preserving fine‑grained training signals.
Each rollout runs as an autonomous loop that owns its environment, drives dialogue generation, and queries the inference engine only for new tokens.
How does the Agent Loop differ from a naïve “run‑once” inference call?
A naïve call would re‑tokenize the entire dialogue each time, discarding cached KV state. The Agent Loop preserves KV across turns, only sending the new suffix, which cuts compute and latency dramatically for long conversations.
Rollout data is stored in a four‑level tree so that credit can be assigned at the granularity that actually matters.
Why not store rollouts as a flat list of tokens?
A flat list would make it impossible to mask or reshape credit for off‑path turns, and KV‑cache reuse would be lost because the system could not identify which prefix belongs to which dialogue branch.
The module isolates infrastructure failures and degenerate turns by masking or reshaping their advantage, ensuring that only useful model behavior drives learning.
Rule detects that Segment 3 triggers a tool‑unavailable error.
Strategy applies a mask to Segment 3, removing it from the loss.
Advantage shaping subtracts a fixed penalty of 0.2 from the advantage of Segment 2.
Early‑stop halts further generation; the Context now contains only Segments 1 and 2.
Since the Context still has a surviving model turn (Segment 2), it is retained for training.
Masking prevents spurious gradients from infrastructure failures, while advantage shaping preserves the relative quality signal of the remaining model turn.
How is the Penalty Module different from simply discarding failed rollouts?
Discarding would lose all useful turns from that rollout. The Penalty Module selectively masks only the offending turn and reshapes advantages, keeping the rest of the trajectory for learning.
The pool groups many persistent host actors, each serving multiple concurrent harness instances and Agent Loops, to avoid exploding file‑descriptor counts.
Why not allocate a dedicated Ray actor per harness?
Each actor would consume a file descriptor on the GCS node; with thousands of concurrent rollouts the descriptor limit would be exceeded, causing crashes. The pool approach amortizes descriptors across many tenants.
Payload Porter splits heavy rollout payloads into lightweight metadata and a stored payload, letting the driver schedule on cheap metadata alone.
How does the Payload Porter avoid a bottleneck when many gigabytes of multimodal data are generated?
By writing only the delta (new images or screenshots) to the store and keeping the bulk of the payload off the driver, the system moves pixels only when the vision encoder needs them, dramatically reducing network traffic.
The Sample Mixer orchestrates four mechanisms that keep the training distribution stable despite heterogeneous rollout durations.
Why not simply increase the number of rollouts for slow sources instead of using adaptive concurrency?
Increasing rollouts without accounting for rollout duration would overload GPU memory and KV capacity, causing severe latency spikes. Adaptive concurrency scales the number of concurrent rollouts with the measured duration $t_i$, keeping resource usage balanced.
**Figure 13** Overview of MiMo MOPD2. (a) Domain-specialized teachers are trained with MixRL on verifiable tasks or with SFT on synthetic demonstrations for open-domain tasks. (b) Standard MOPD uses RL teachers to supervise full student rollouts. (c) Prefix-Conditioned OPD reuses trajectories from teacher rollouts (Teacher-Prefix OPD) or SFT data (SFT-Prefix OPD). A source trajectory with $k$ assistant-turn decision points yields $k$ complete history prefixes $h_i$, each of which can initialize a separate student-generated turn $y_i$ for token-level distillation by the relevant domain teacher. SFT data provide prefix contexts rather than fixed continuation targets.
**Figure 14.** Overview architecture of RL infrastructure.
**Figure 15** Rollout heterogeneity across 25 data sources. Each line connects one source's *Start* point (faint) to its *End* point (solid); each point plots that source's mean generated tokens against mean rollout time over completed rollouts (tokens summed across dialogue contexts before context filtering). Both axes are logarithmic.
**Figure 16** Trace-driven scheduling simulation for six sources. Legend: mean duration / group acceptance rate. Top: Collection progress (accepted groups / nominal per-step targets, with surplus carried forward). Bottom: Rollout occupancy (shares of occupied sequence slots). Step spacing reflects elapsed time. Dotted lines mark step boundaries. Training time, credit-assignment latency, staleness expiry, and replay are excluded.
Open Foundations
We ablate core pipeline components to measure their effect on rollout occupancy, speed, and final performance.
We systematically remove or modify individual ingredients of the RL pipeline and report the resulting changes in rollout occupancy, training wall‑clock time, and downstream task scores.
Using a deficit‑based $\alpha$=0 reduces rollout occupancy by 35.9 % compared with the target‑based $\alpha$=1 baseline.
Occupancy drops from 100 % ($\alpha$=1) to 64.1 % when $\alpha$=0.
Deficit‑based $\alpha$=0 also increases the per‑step training wall‑clock time by 4.3 minutes.
Training time rises from 12.1 min ($\alpha$=1) to 16.4 min ($\alpha$=0).
Deficit‑corrected $\alpha$=0.5 further lowers occupancy to 61.5 %, a 38.5 % drop relative to the $\alpha$=1 baseline.
Occupancy 61.5 % versus 100 % for $\alpha$=1.
The same $\alpha$=0.5 setting adds 2.1 minutes to the training step.
Training time 14.2 min versus 12.1 min for $\alpha$=1.
Initializing the system with a steady‑state startup ($\alpha$=0.5) yields an intermediate occupancy of 89.0 %, only an 11.0 % loss.
Occupancy 89.0 % compared to 100 % for the target baseline.
Steady‑state startup also inflates the training step to 50.7 minutes, a 38.6‑minute increase.
Training time 50.7 min versus 12.1 min for the target baseline.
Switching from block‑8 to block‑6 DFlash improves global average throughput by roughly 6 % without harming acceptance length.
Throughput gain measured on mixed‑task RL evaluation; acceptance length unchanged.
Applying FP8 low‑precision to the draft stage adds a further 10.3 % per‑node throughput over the FP16 baseline.
Per‑node throughput measured on the long‑context workload.
Training with the multi‑harness RL scheme improves coding benchmarks by up to 9.3 percentage points over single‑harness RL.
MiMo Code Bench (mini) gains range from +1.8 pp to +9.3 pp across seven harnesses (Table 7).
Conclusion
The conclusion recaps MiMo‑V2.6 advances, releases resources, and acknowledges contributors.
We introduced the MiMo‑V2.6 series as a practical pathway to scale foundation models via large‑scale agentic reinforcement learning.
Scaling was achieved along three axes: increasing training batch size and throughput, expanding the diversity and complexity of environments and agent harnesses, and allocating more compute to groupwise agentic grading.
Key infrastructure components—asynchronous training, mixed‑task rollout pipelines, training‑inference consistency mechanisms, and safeguards against drift and reward hacking—made sustained long‑horizon optimization possible while improving solution quality and token efficiency.
Evaluations on public and internal benchmarks show MiMo‑V2.6 attaining competitive performance across a broad spectrum of agentic tasks.
To foster community progress we release MiMo‑V2.6‑Distill‑Qwen‑9B together with curated task environments, verifiers, an end‑to‑end RL framework, and composable mini‑harnesses.
Consistent RL gains across domains and harnesses demonstrate the value of these resources as a shared foundation for further experimentation.
We emphasize that broad exploration, informative feedback, and scalable training systems are jointly essential for advancing model self‑improvement.
We hope the open resources support reproducible research and cumulative progress toward increasingly capable, general‑purpose agents.
We thank all contributors—including the Xiaomi Data Platform, CloudML, NGK, MiChat, Mify, MiKS, and LLM‑Plus teams—and list the core contributors in reverse alphabetical order by first name.
A complete bibliography of cited works follows.