Miles v0.1: Production-Level Post-Training
RadixArk, Tom Chen, Mao Cheng, Shi Dong, Kangrui Du, Yanbin Jiang, Jiajun Li, Yiming Li, Tao Lin, Yusheng Su, Andy Ye, Yueming Yuan, Zhichen Zeng
Miles v0.1 is a production-ready, asynchronous RL system for frontier-scale models that decouples rollout generation from training.
How can we architect a production-ready, full-stack reinforcement learning system that decouples rollout generation from weight updates to maximize throughput and fidelity?
Frontier-scale reinforcement learning (RL) often forces a choice between idle hardware and training instability: synchronous loops leave GPUs waiting for slow rollouts, while asynchronous systems suffer from "train-rollout mismatch" where the trainer and rollout engines diverge numerically. Miles v0.1 solves this by enforcing a strict "token-in-token-out" (TITO) contract and expert-routing replay, ensuring the trainer sees the exact tokens and expert assignments the policy sampled. In a case study on a 744B-parameter model, the system maintained a median step time of 263 seconds across 64 NVIDIA GPUs while running fully asynchronous agentic RL.
Paper Primer
Miles treats the rollout engine and trainer as independent, disaggregated pools that communicate via a bounded buffer. To prevent the numerical drift that typically plagues asynchronous RL, it uses a session server that forces the rollout engine to record the exact token IDs and expert-routing decisions, which the trainer then replays during the forward pass.
Miles eliminates silent train-rollout divergence in MoE models.
By recording expert-routing decisions during rollout and replaying them during training (R3), the trainer avoids the stochastic expert-selection mismatch that destabilizes RL in large Mixture-of-Experts models. The system maintains a 96% prefix-cache hit rate for multi-turn agentic rollouts using session-aware routing.
The system enables fully asynchronous training without stalling.
By decoupling generation and training into separate GPU pools and using a bounded buffer to absorb rate differences, the system avoids the idle time inherent in synchronous turn-taking. Median step time of 263 seconds for a 744B-parameter model on 64 GPUs.
Why does Miles require a "session server" instead of letting the rollout engine talk directly to the trainer?
The session server acts as the authoritative source of truth for tokenization and message history. By controlling the tokenization process, it ensures the trainer reconstructs the exact sequence sampled by the policy, preventing the silent token-level mismatches that occur when chat templates or tool-call parsers are applied inconsistently.
What is the primary trade-off when enabling Rollout Routing Replay (R3)?
R3 requires storing and moving large routing tensors alongside each trajectory. For a 32K-token sequence over 60 layers, this adds roughly 60MB of overhead per trajectory, which necessitates sufficient memory headroom in the training actor.
System Overview and Motivation
We expose the core design that separates rollout from training to boost RL system throughput.
Frontier‑scale reinforcement‑learning post‑training struggles to keep rollout latency and training throughput aligned, and the numerical drift between the two stages can invalidate the learning objective.
The system is split into two loosely coupled pipelines—Rollout and Trainer—so each can run at its own pace while periodically exchanging weights.
Slime is a minimal, composable framework that enforces component verification and clean interfaces, serving as the blueprint for Miles.
The figure illustrates the architecture of a reinforcement learning system, divided into two main components: "Rollout" and "Training". The Rollout section contains "SGLang engines", a "TITO session" for exact token IDs, and "Agents & environments". The Training section includes a "Training backend" (Megatron or FSDP), "RL loss" (GRPO, GSPO, PPO, and more), and an "Optimizer" that is offloadable to CPU or NVMe. Data flows from the Rollout section to the Training section via a "data buffer" labeled with "trajectories" and "staleness, sampling policy, replay data". A dashed line indicates a "weight update" from the Training section back to the Rollout section, utilizing "NCCL broadcast, P2P RDMA, or CUDA IPC when colocated".
Decoupling rollout and training is the core design principle that enables high‑throughput, low‑latency RL at frontier scale.
Rollout Generation Mechanics
How Miles decouples rollout from training to keep both fast and accurate.
Rollout generation dominates wall‑clock time, so the system must keep the engines busy while preserving the exact tokens the policy sampled.
By separating rollout from the trainer, keeping each trajectory’s KV cache on a single engine (affinity), and buffering completed groups, Miles can generate data continuously without sacrificing the exact token sequence the trainer later consumes.
How does this rollout buffer differ from a simple FIFO queue?
The buffer not only queues groups but also applies three explicit discard checks—generation failure, user‑defined filtering, and staleness relative to the trainer’s current weight version. A plain FIFO would keep every finished group, eventually feeding stale or invalid data to the trainer.
When trajectory A₁ finishes, its slot becomes free; the buffer immediately starts a new trajectory A₁′, keeping three active groups.
Group B finishes both trajectories; both slots are freed, and two new trajectories B₁′ and B₂′ are launched, restoring the buffer to its capacity of 4 groups.
The trainer pulls the oldest finished group (C) as a batch of 2 trajectories; the buffer now holds groups A, B, and the newly launched A₁′, B₁′, B₂′.
Later, the trainer’s weight version advances to v₅ while group A still contains a trajectory generated with weight v₃; the staleness check (v₅ − v₃ = 2) exceeds the configured limit, so group A is discarded.
The buffer’s discard logic prevents the trainer from learning from outdated weights, even though the rollout engines keep producing new data.
Router receives a new session request, extracts its stable routing key, and assigns the session to the least‑loaded engine (affinity).
Engine generates the first turn, stores the KV cache, and returns the token IDs to the session server.
For each subsequent turn, the router routes the request to the same engine; only the new suffix is generated and appended to the cached prefix.
When a trajectory finishes, the engine puts its group into the bounded buffer.
The trainer periodically pulls the oldest finished groups that pass the three discard checks.
Trainer updates the model weights; the new weight version is broadcast to rollout engines, which pause briefly to install the update.
**Table 1.** The three reasons the buffer drops a finished group. The first two are properties of the group, so they are checked as soon as it arrives; staleness depends on how long the group waited, so it is checked only when the trainer collects it. The run supplies the filter and the staleness limit and chooses whether dropped prompts are retried or discarded, except that filter-rejected groups are always discarded because they carry no gradient signal.
**Table 2.** Buffer metrics reported on every training step, under the `rollout/fully_async/` prefix. Staleness appears twice: once for the groups a step drew, and once for the groups still waiting.
**Table 3.** The three evaluation modes under fully asynchronous rollout. Shared engines measure whichever weights the fleet most recently received, and new generation stops while they do so, although in-flight requests finish. The snapshot-based modes measure the exact weights in the snapshot they receive; an external backend sees only a checkpoint directory, so it can be any service, with or without SGLang.
Environment Integration and Buffering
How Miles integrates diverse environments, preserves token fidelity, and replays expert routing for reliable RL.
Rollout‑trainer decoupling exposes two failure modes: token‑level drift between the rollout engine and the trainer, and expert‑routing drift in mixture‑of‑experts models. Both break the importance‑sampling ratio that RL relies on, causing policy collapse.
To let arbitrary RL environments plug into Miles, the system defines three nested layers—agent, generate, and rollout—each exposing a distinct responsibility boundary.
TITO guarantees that the trainer sees exactly the token IDs produced by the rollout engine, eliminating the token‑mismatch that plagues multi‑turn pipelines.
Turn 1: server tokenizes “Hello” → IDs [101, 102]; checkpoint C₁ stores [101, 102].
Turn 2: server reuses checkpoint C₁, tokenizes only the suffix “world!” → IDs [103, 104]; checkpoint C₂ stores [101, 102, 103, 104].
Trainer reads the concatenated IDs [101, 102, 103, 104] and the associated log‑probabilities, guaranteeing perfect alignment with rollout.
The trainer can now compute loss on the exact sampled tokens, so the importance ratio stays near 1.
Two policies—linear and branching—determine how a new request can extend a stored session history.
Linear rule: server checks deepest prefix (A→B). Since D diverges from C, the request is rejected.
Branching rule: server attaches D as a new child of node $B$, creating branch B→D while preserving original branch B→C.
Result: linear session yields one trajectory (A→B→C); branching session yields two trajectories (A→B→C and A→B→D).
Branching enables harnesses that fork or compress context, at the cost of storing multiple leaf trajectories.
When extending a session, the server must decide whether a stored message matches the incoming request; three configurable policies trade strictness for flexibility.
Before a checkpoint can be used with TITO, Miles verifies that its tokenization remains append‑only across turns.
R3 records the exact expert assignments for each token during rollout and replays them during training, eliminating routing drift.
Each integer occupies 4 bytes → total bytes = 31 999 × 60 × 8 × 4 ≈ 61 MB.
Routing tensor is kept in memory for the entire trajectory and streamed together with token IDs.
During training, the trainer reads the stored expert IDs and routes each token accordingly, bypassing the router.
Even though the overhead is sizable, it is affordable within the memory headroom described in Section 3.2, and it prevents catastrophic policy drift in MoE models.
**Figure 2.** The three evaluation modes on the training timeline. Sharing the rollout engines halts generation while the evaluation runs. The two snapshot-based modes run alongside training, although exporting a fresh snapshot or having too many evaluations already outstanding can still pause the trainer.
**Figure 3.** Token-in, token-out. The session server preserves the exact token IDs produced by the engines, so the trainer sees the model-generated tokens even when the harness is a black box.
Trainer Design and Precision
Training details precision, memory, backend, and objective choices for scalable RL.
The trainer consumes rollout data to produce weight updates, and four properties—numerical precision, memory footprint, backend choice, and objective—determine whether a run can scale to frontier‑size models.
Both rollout and trainer must apply the exact same quantization to shared weights, otherwise their computations diverge layer by layer.
Compute the integer code for each entry: $\text{code}= \text{round}(W/s) = \begin{bmatrix}0 & -1\\ 2 & 3\end{bmatrix}$.
Clamp codes to the representable range $[-4,3]$ (E4M3 limits).
Reconstruct the quantized matrix $\hat{W}=s\cdot\text{code}= \begin{bmatrix}0.00 & -0.25\\ 0.50 & 0.75\end{bmatrix}$.
Both rollout and trainer use $\hat{W}$ for their forward passes, guaranteeing identical activations.
The contract’s bit‑exact quantizer ensures that any downstream error originates from the model itself, not from mismatched precision between stages.
Why can’t we simply quantize the trainer while keeping rollout in BF16?
Quantizing only the trainer would make its forward pass use different numeric values than rollout, breaking the assumption that both sides evaluate the same policy. The contract forces both sides to share the same quantized weights, preserving the correctness of the importance‑ratio computation.
**Table 5.** Low-precision formats with an end-to-end Miles recipe, against the BF16 baseline. NVFP4 nests an E4M3 scale per block inside one FP32 scale per tensor. FP8 blockwise runs on NVIDIA Hopper and Blackwell and on AMD MI350X and MI355X; MXFP8 and NVFP4 need Blackwell; A100 has no FP8 arithmetic and runs BF16 only.
When the training actor is idle, its entire state can be moved off the GPU so the rollout engine can reuse the same memory.
The actor’s state is serialized into a single $200\,$MiB buffer.
The buffer is copied via PCIe into a pinned host buffer, completing in $\approx0.2\,$s.
When the rollout engine needs the GPUs, the buffer remains on host memory, freeing the entire $16\,$GiB of GPU memory.
Upon resumption, the buffer is copied back to the GPU in the same $0.2\,$s window.
Because the whole state moves as one block, the offload/reload latency is independent of the number of tensors, making it scalable to very large models.
How does offloading differ from simply reducing the batch size to fit in memory?
Reducing batch size lowers compute throughput and can degrade learning dynamics, whereas offloading preserves the original batch size and only moves idle state, keeping the training dynamics unchanged.
Instead of keeping the full Adam moments on the GPU, Miles streams only the bucket needed for the current parameter update.
For bucket 1, the corresponding optimizer‑state file (≈80 MiB) is read from SSD into a pinned GPU buffer.
The forward and backward passes compute gradients for the $10^7$ parameters.
The Adam update uses the loaded moments, writes the updated moments back to the same buffer, then the buffer is flushed to disk.
Buckets 2–100 repeat the same sequence, never holding more than one bucket in GPU memory.
Only $80\,$MiB of optimizer state resides on the GPU at any time, allowing the rest of the $8\,$GiB to be freed for model activations.
Why does streaming the optimizer state not affect the final model quality?
Because the optimizer moments are read, updated, and written back without alteration; the only change is the timing of their presence in GPU memory, which does not modify the mathematical update.
Megatron‑LM and FSDP provide two different ways to partition the model across GPUs, each with its own trade‑offs.
When would a researcher prefer FSDP over Megatron‑LM despite the latter’s richer parallelism?
If the model fits within a single data‑parallel shard and the researcher wants to load a Hugging Face checkpoint without extra conversion steps, FSDP offers a faster path to a first run and easier debugging.
Because rollout and trainer use different kernels and precisions, their policy probabilities differ; corrections clamp or drop extreme importance‑ratio values.
Why does Clip‑or‑Pop drop tokens instead of scaling them like TIS?
Dropping tokens removes their contribution entirely, which can be preferable when an outlier ratio indicates a fundamentally unreliable sample; scaling would still inject noisy gradient information.
Together, low‑precision contracts, memory‑offloading strategies, backend selection, and ratio corrections form the training stack that lets Miles scale RL to frontier‑size models while keeping throughput and numerical stability under control.
Weight Synchronization and Handoff
Weight synchronization contracts define how trainers hand off updated weights to rollout engines efficiently.
When trainer and rollout fleets run on separate GPUs, moving updated weights can dominate the training step. A naïve per‑tensor copy would stall the pipeline, especially at frontier scales where a full NCCL broadcast can take nearly a minute.
The trainer prepares a bucket of converted weights, hands it off, and rollout engines start generating only after the bucket is flushed.
Step 1: $W_1$ is gathered, converted to Hugging Face layout, and appended to the buffer (buffer usage = 120 MB).
Step 2: $W_2$ is gathered and appended (buffer usage = 256 MB).
Step 3: Buffer is still below capacity, so the trainer continues gathering other tensors.
Step 4: After $W_3$ (300 MB) is added, the buffer exceeds 512 MB, triggering a flush.
Step 5: The entire 512 MB buffer is handed to the selected transport; rollout engines receive the new weights atomically.
The contract guarantees that no rollout engine sees a partially updated model, eliminating race conditions that would otherwise corrupt inference.
How does this contract differ from a traditional parameter‑server update?
Parameter‑server updates push individual parameter shards asynchronously, so different workers can observe different versions. The weight‑synchronization contract batches all shards into a single bucket and blocks rollout until the whole bucket is applied, ensuring a globally consistent policy.
Gather tensor‑parallel shards for each weight tensor and convert them to the canonical Hugging Face naming scheme.
Append the converted tensors to a fixed‑size buffer (default 512 MB).
When the buffer fills, invoke the selected transport’s flush operation.
For broadcast: issue an asynchronous NCCL broadcast of the entire buffer to all rollout ranks.
For peer‑to‑peer: write each shard directly into target memory over RDMA, using a pre‑computed transfer plan.
For disk‑delta: write only the changed bytes to a shared filesystem; rollout hosts patch their local checkpoints.
After the flush completes, release the lock, and rollout engines resume generation under the new policy.
**Table 7.** The three weight-synchronization transports. All deliver the same converted weights but differ in the connectivity they assume and in how transfer volume scales with the fleet. When training and rollout are colocated on the same GPUs, the handoff is local and no transport is involved.
**Table 8.** Time per weight update, P2P against NCCL broadcast, on H100 clusters with a 1 GB transfer bucket, averaged over steady-state steps and timed from the end of the generation pause to the return of the update call. Node counts are per side, with trainer and rollout fleets of equal size. The Kimi K2 times include about 884 ms of on-GPU requantization that its checkpoint requires after every transfer. The advantage grows with fleet width rather than model size, and appears already at two nodes per side.
After weights are synchronized, Miles can pause generation, verify that every rollout engine holds the exact version, and then resume. This check guarantees bit‑exact equality (or a quantization‑aware tolerance) before any new requests are processed.
Alternative Training Recipes
Three lightweight recipes modify the Miles RL loop while keeping its core structure.
The Miles loop separates rollout, training, and weight‑update stages; this section shows how each stage can be swapped out without breaking the overall pipeline.
On‑policy distillation cuts response length by 56 % while leaving accuracy statistically unchanged.
In the Qwen3.5‑35B‑A3B run, average tokens per response drop from 14,070 to 6,132 and accuracy moves from 84.0 % to 85.2 % (standard error ≈ 1.6 %).
LoRA RL replaces full‑parameter updates with low‑rank adapters; only the adapters are gradient‑computed, synchronized, and served, which trims both compute and communication overhead.
**Table 6.** The two training backends. Megatron-LM exposes model-parallel axes, while the current FSDP backend uses data-parallel sharding; a $\times$ marks what that backend does not yet implement, not a limit of FSDP itself.
**Figure 4.** On-policy distillation. The student generates a response, the teacher scores the same tokens position by position, and the difference between the two log-probabilities, a per-token reverse-KL estimate, drives the update, optionally alongside a task reward.
True‑on‑policy alignment achieves exactly zero log‑probability difference between rollout and trainer, at the cost of reduced throughput.
In the Qwen3‑4B‑Base run, the reported absolute difference between the two engines is 0, while the rollout latency increases relative to the baseline.
These recipes demonstrate that the Miles loop can be re‑wired for non‑standard RL tasks without redesigning the whole system.
Diffusion Model Alignment
Diffusion RL gains speed and precision controls across multiple recipes.
Miles‑Diffusion extends the rollout‑trainer decoupling to image and video diffusion models, enabling RL‑aligned generation at higher throughput.
Miles‑Diffusion cuts rollout time per step from 157.4 s to 87.6 s, a 44% reduction, and lowers total step time from 321.9 s to 252.1 s.
Measured on the LTX‑2.3 recipe after introducing deserialization and worker‑pool optimizations.
Ensures that forward and backward passes produce identical floating‑point results across runs, eliminating stochastic noise in the RL update.
How does deterministic mode differ from merely fixing random seeds?
Fixing seeds only controls the order of random number generation; deterministic mode also patches low‑level kernels so that floating‑point arithmetic yields the same bit pattern on identical inputs, removing nondeterminism from the compute path itself.
Keeps precision‑sensitive parameters (e.g., timestep embeddings) in FP32 while the bulk of the model runs in BF16, narrowing the training‑inference gap.
Why not keep the entire model in FP32 for safety?
Running the whole model in FP32 would roughly double memory usage and halve throughput, making large‑scale diffusion training impractical; the per‑parameter approach retains accuracy where it matters while still leveraging BF16 efficiency.
Recipes are classified by verification level: Fully gated (complete training curve + deterministic test), Proxy gated (scaled‑down proxy test), Verified (training curve only), and Not verified (no full curve).
On the current main branch, SD3.5 and LTX‑2.3 are Fully gated; Wan2.2 is Proxy gated; Qwen‑Image and Cosmos 3 are Verified; the two Wan2.2 LoRA variants remain Not verified.
Model Coverage and Support
Six frontier models are fully supported on release‑day rollout.
Miles and SGLang support six frontier models on day‑0 release.
Kimi K3, DeepSeek‑V4, GLM‑5.2, Qwen3.8, Inkling, and NVIDIA Nemotron 3 Ultra are all runnable immediately after their weights become public.
Beyond the release‑day set, Miles documents recipes for nine model families—dense, MoE, hybrid‑attention, and multimodal—but the level of support varies from full‑scale validated runs to reduced‑layer CI tests.
Hardware coverage spans NVIDIA A100‑GB300 GPUs and AMD MI350X‑MI355X Instinct GPUs, with each platform requiring a matching numeric format (BF16 on A100, MXFP8/NVFP4 on Blackwell, FP8 on Hopper and supported AMD GPUs).
End-to-End Case Study: GLM-5.2
End‑to‑end RL run on GLM‑5.2 demonstrates the system’s throughput and fidelity.
Miles v0.1 separates rollout from training, letting the two stages run asynchronously while keeping weights synchronized.
Median training step time is 263 seconds.
Figure 5 panel (a) shows the median of the first 30 measured steps after clipping the warm‑up outlier.
**Table 9.** Configuration of the GLM-5.2 agentic RL reference run.
**Figure 5.** Metrics from the GLM-5.2 agentic RL reference run. (a) wall-clock seconds per training step for the first 30 measured steps; the step-0 warm-up value is clipped, and the median is marked. (b) the divergence between the rollout engine’s and the trainer’s log-probabilities at the sampled tokens, with the mean marked. (c) raw task reward, with a faint line for each step and a bold line for its nine-step moving average.