The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning
Jing Liang, Hongyao Tang, Yi Ma, Yancheng He, Weixun Wang, Xiaoyang Li, Ju Huang, Wenbo Su, Jinyi Liu, Yan Zheng, Jianye Hao, Bo Zheng
MIPU stabilizes LLM reinforcement learning by optimizing for inference-side performance rather than training-side surrogates.
How can we align LLM reinforcement learning updates with actual inference performance to prevent training collapse?
Modern LLM reinforcement learning pipelines often use separate engines for rollout generation and gradient computation, creating a "training-inference mismatch" where the policy used for deployment differs from the one being updated. This discrepancy means that an update appearing beneficial to the training engine can actually degrade the inference policy used in production. The authors propose Monotonic Inference Policy Improvement (MIPI), a principle that shifts the optimization objective from the training policy to the inference policy. They implement this via Monotonic Inference Policy Update (MIPU), a two-step framework that constructs sampler-referenced candidate updates and selectively accepts them based on an inference-side gap proxy. Experiments under high-mismatch FP8-quantized rollouts show that MIPU achieves higher average reasoning accuracy and more stable training dynamics than standard reinforcement learning baselines.
Paper Primer
MIPU realizes the MIPI objective by decomposing the inference-policy improvement into two distinct phases: sampler-referenced candidate construction and inference-gap-aware acceptance. The core move is to treat the training-side update as a proposal that must pass a validation test before being synchronized to the inference engine.
Step 1 generates a candidate policy using a truncated importance-sampling correction that aligns the training update with the sampler's distribution. Step 2 then evaluates the synchronized candidate using a post-update gap proxy: if the inference engine's performance on validation data suggests the update is unreliable, the system rejects the candidate and rolls back to the previous checkpoint.
MIPU improves reasoning performance and training stability under high training-inference mismatch.
Evaluations on Qwen-3-4B and Qwen-3-1.7B across five mathematical benchmarks (e.g., MATH-500, AIME24) show consistent gains over standard GRPO and mismatch-correction baselines. MIPU reached 66.71% average pass@1 accuracy on Qwen-3-4B, outperforming baselines that frequently suffered from training collapse.
Ablation studies confirm that Step 1 and Step 2 are complementary: Step 1 improves the quality of candidate updates, while Step 2 prevents the accumulation of updates whose gains are not realized by the inference engine. Random rollback controls demonstrate that Step 2's stability is due to its inference-gap signal, not merely the reduction in update frequency.
Why does standard reinforcement learning for LLMs fail when training and inference engines are mismatched?
Standard algorithms optimize the training policy, but deployment relies on the inference policy. When these policies diverge, an update that appears to improve the training-side surrogate can actually lead to a worse inference policy, causing instability or performance collapse.
How does this approach differ from existing infrastructure-level remedies for mismatch?
Infrastructure methods attempt to reduce the discrepancy between engines (e.g., using FP16 rollouts). MIPU is an objective-level framework that accepts that mismatch exists and uses a validation-based acceptance test to ensure only beneficial updates are deployed.
The Mirage of Training Optimization
LLM RL updates a training policy that can diverge from the deployed inference policy, causing instability.
LLM reinforcement learning updates the training policy $\\pi$, but the inference policy $\\mu$ used for generation often differs because rollout and gradient computation run on separate engines with distinct precision, decoding, or quantization. This training‑inference mismatch means that a better $\\pi$ does not automatically yield a better $\\mu$, leading to instability and occasional collapse.
The model is optimized with one policy but deployed with another, so gains seen during training may not appear at inference time.
The failure of training‑side objectives to guarantee inference‑side improvement is the core problem this work resolves.
Reinforcement Learning for LLMs
We review RL fundamentals, policy mismatch, and the PPO‑IS baseline.
Reinforcement Learning (RL) provides a general framework for sequential decision‑making, and it has become the dominant paradigm for fine‑tuning large language models (LLMs). In this setting the interaction is modeled as a Markov Decision Process (MDP) $M=(S,A,P,R,\gamma)$, where a state $s_t=(q,y_{1:t})$ encodes the prompt $q$ together with the tokens generated so far, and an action $a_t\in A$ selects the next token from the vocabulary $V$.
The training pipeline uses the policy $\pi$ (the LLM parameters during gradient updates) while the deployed engine samples from an inference policy $\mu$, which may be quantized or sampled differently. This training‑inference mismatch—$\pi\neq\mu$ despite sharing parameters—creates non‑stationarity and suboptimal rollouts, motivating methods that align the two policies.
PPO‑IS is the standard proximal policy optimization algorithm that corrects for distribution shift between the behavior policy and the target policy using importance‑sampling ratios.
Monotonic Policy Improvement seeks a new policy $\pi_{k+1}$ such that $J(\pi_{k+1})-J(\pi_k)\ge0$, guaranteeing non‑decreasing expected reward. Directly optimizing this is intractable, so practical methods replace the unknown visitation distribution of $\pi_{k+1}$ with that of $\pi_k$ and form a surrogate $\tilde{\Delta}$ that can be estimated via importance sampling.
Monotonic Inference Policy Update
MIPU enforces monotonic improvement of the inference policy through a two‑step sampler‑referenced update and gap‑aware acceptance.
When the training policy $\pi$ improves while the inference policy $\mu$ stays misaligned, a nominal training gain can actually degrade the model’s deployed behavior.
**Figure 1.** Monotonic Inference Policy Update (MIPU) resolves the *Objective Misalignment* issue of LLM RL. Canonical LLM RL accepts synchronized updates by a training-side objective, which does not necessarily imply improvement of the inference policy. Here, $\pi$ and $\mu$ denote the training policy and inference policy respectively, $c$ is a tolerance parameter accounting for proxy noise. To address this mismatch, we propose a new principle as monotonic improvement on the inference-policy trajectory (the MIPI principle). MIPU realizes this principle with two steps: Step 1 optimizes Terms ②+③, while Step 2 estimates and validates Term ①, jointly covering all three terms in the MIPI decomposition.
MIPU guarantees that each accepted update improves the inference‑policy objective, not just the training‑side surrogate.
Compute the pre‑update mismatch weights wₖ^{(i)} = $\pi$ₖ($y_i$)/$\mu$ₖ($y_i$): w₁ = 0.5/0.6 ≈ 0.83, w₂ = 0.5/0.4 = 1.25.
Apply the truncation \bar wₖ^{(i)} = min(wₖ^{(i)}, $w_{max}$^*) with $w_{max}$^* = 1.0, yielding \bar w₁ = 0.83, \bar w₂ = 1.0.
Form the sampler‑referenced surrogate (Equation 7) using the current‑update ratios $r_i$($\theta$)≈1 (no change) and the advantages: contribution₁ = 0.83 × 1 × 1.0 = 0.83, contribution₂ = 1.0 × 1 × (‑0.5) = ‑0.5.
Sum gives a surrogate gain of 0.33, so the optimizer proposes an updated trainer $\pi_{k+1}$ that slightly boosts the probability of y₁.
Synchronize to inference $\mu_{k+1}$ (e.g., by copying parameters). Compute the post‑update gap proxy (Equation 8); suppose it yields $T_{post}$ = ‑0.02.
With tolerance c = 0.05, the gap passes (‑0.02 ≥ ‑0.05), so the update is accepted.
The truncation prevents a large pre‑update mismatch (w₂ = 1.25) from overwhelming the gradient, while the gap test catches cases where the candidate would hurt the deployed inference policy.
How does MIPU differ from the standard PPO‑IS variant that simply clips the trainer‑to‑sampler ratio?
Standard PPO‑IS clips the full ratio $\rho$ᵢ = $\pi$_$\theta$/$\mu$ₖ, which mixes the pre‑update mismatch wₖ^{(i)} = $\pi$ₖ/$\mu$ₖ with the current‑update ratio rᵢ($\theta$). MIPU isolates the current update by clipping only rᵢ($\theta$) and applies a capped correction \bar wₖ^{(i)} to the mismatch, thereby avoiding over‑constraining the step when the old policies are far apart.
Initialize trainer $\pi_{θ0}$ with a base model and set inference $\mu$₀ = Sync($\pi_{θ0}$).
For each iteration k, save a checkpoint of ($\theta_{k}$, optimizer state, $\mu_{k}$).
Collect a training batch and G samples per prompt from the current inference sampler $\mu_{k}$.
Compute group‑relative advantages \hat $A_{\mu_k}$ and mismatch weights $w_k$ = $\pi_{k}$/$\mu_{k}$.
Apply truncation \bar $w_k$ = min($w_k$, $w_{\text{max}$}^*) and clip only the current‑update ratios $r_i$($\theta$) in the surrogate $J_{S1}$ (Equation 7) to obtain the candidate trainer $\pi_{k+1}$.
Synchronize the candidate to inference $\mu_{k+1}$ = Sync($\pi_{k+1}$).
Sample a validation batch from $\mu_{k+1}$, compute validation advantages \hat $A_{\mu_{k+1}$} and length‑normalized importance weights $\rho_{i}$ (Equation 9).
Estimate the post‑update gap $T_{post}$ using the proxy (Equation 8).
If $T_{post}$ < ‑c, reject the update and restore the saved checkpoint; otherwise accept and continue.
Empirical Evaluation and Stability
MIPU’s accuracy and stability under FP8 rollout, plus ablations and analysis.
The paper’s core claim is that the inference policy can drift away from the training policy when quantized rollout is used, and MIPU keeps them aligned.
MIPU achieves the highest average pass@1 accuracy under FP8‑quantized rollout.
66.71 % on Qwen3‑4B and 53.97 % on Qwen3‑1.7B, surpassing all baselines.
During generation the model’s activations are cast to 8‑bit floating point, so the sampled tokens come from a lower‑precision inference policy.
How does FP8‑quantized rollout differ from the more common FP16 rollout?
FP8 halves the mantissa width compared with FP16, so rounding errors are larger and the sampled token distribution shifts more dramatically, which makes the inference‑policy $\mu$ diverge faster from the training‑policy $\pi$.
**Figure 5.** Step 1 implementation analysis under the Qwen3-4B FP8-quantized rollout. Comparison of PPO-IS, Vanilla-IS, and TIS in terms of performance, gradient norm, inference-training K3-KL, and clip ratio.
**Figure 6.** Sensitivity to the acceptance tolerance $c$ under the Qwen3-4B FP8-quantized rollout in terms of inference-training K3-KL, training score, and $\widehat{T}_{\text{post}}$.
**Figure 2.** Performance of different methods under FP8-quantized rollout. Compared methods show unstable training dynamics and may suffer from sharp performance drops, while MIPU maintains a stable score trajectory.
**Figure 3.** Training curves for ablation studies under FP8-quantized rollout. We show the training score, the inference-training K3-KL, $\widehat{T}_{post}$ (i.e., inference gap) and the rollback rate computed over a 100-step moving window. Step 1 improves the candidate update direction, while Step 2 introduces inference-gap-aware acceptance to filter unreliable synchronized candidates. The full method obtains stronger performance with a more controlled inference-policy trajectory.
**Figure b.** Step 2 vs. random rollback.
**Figure a.** Inference-training K3-KL and inference gap.
$T_{\text{post}}$ estimates how much the candidate update will misalign the inference policy after the rollout.
Implementation and Hyperparameters
Implementation specifics, hyperparameters, and additional analyses.
A.1 lists the three groups of hyperparameters that define decoding, training, and infrastructure settings. The values (e.g., temperature 1.0 for training, learning rate 1e‑6, 8 × H100 GPUs) are summarized in Table 3.
**Table 3.** Hyperparameter setups.
A.2 describes the baseline (vanilla GRPO with dual‑clipped loss) and the proposed modifications for Step 1 and Step 2, followed by the MIS variant and the learning‑rate schedule.
B analyzes Step 1, showing that clipping the raw ratio $\pi_{\theta}/\mu_k$ (PPO‑IS) mixes pre‑update mismatch with the current update, leading to overly conservative steps. Factoring the ratio into $\frac{\pi_k}{\mu_k}\times\frac{\pi_{\theta}}{\pi_k}$ isolates the mismatch correction and yields two variants: Vanilla‑IS (unbounded correction) and TIS (truncated correction).
C studies the tolerance parameter $c$ in Step 2. Allowing a modest positive $c$ early on lets the method accept useful candidates despite the initially low post‑update gap $T_{\text{post}}$, while annealing $c$ later enforces stricter non‑regression.
D lists the software and dataset licenses that underpin the experiments, ranging from MIT and Apache‑2.0 to assets with no license.