The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning

Jing Liang, Hongyao Tang, Yi Ma, Yancheng He, Weixun Wang, Xiaoyang Li, Ju Huang, Wenbo Su, Jinyi Liu, Yan Zheng, Jianye Hao, Bo Zheng

MIPU stabilizes LLM reinforcement learning by rejecting updates that fail to improve the inference policy.

How can we align LLM reinforcement learning updates with the actual inference policy to prevent training instability and collapse?

Modern LLM reinforcement learning pipelines use separate engines for training and inference, causing the model to behave differently in deployment than it does during gradient updates. This training-inference mismatch means that an update appearing beneficial to the trainer can actually degrade the model's performance when synchronized to the inference engine. The authors propose Monotonic Inference Policy Improvement (MIPI), a principle that shifts the optimization objective from the training policy to the inference policy. They implement this via Monotonic Inference Policy Update (MIPU), a two-step framework that first generates a sampler-referenced candidate update and then selectively accepts it only if it passes an inference-side gap test. Experiments under high-mismatch conditions show that MIPU consistently outperforms standard reinforcement learning baselines in both reasoning accuracy and training stability.

Paper Primer

The core mechanism of MIPU is a two-step filter: Step 1 constructs a candidate update using sampler-referenced importance weights to align the trainer with the inference engine, while Step 2 acts as a gatekeeper. The gatekeeper is like a quality-control inspector: it evaluates the synchronized candidate on a validation set and rolls back the model if the inference-side performance gain is negative.

MIPU improves reasoning performance and training stability under high training-inference mismatch.

Evaluations on Qwen3-1.7B and Qwen3-4B models using FP8-quantized rollouts across five mathematical benchmarks (e.g., MATH-500, AIME24). MIPU achieves the highest average pass@1 accuracy compared to standard GRPO and existing stabilization baselines like MIS and learning-rate decay.

Why does the standard reinforcement learning objective fail in this context?

Standard methods optimize the training policy, but because of implementation differences (like quantization or precision), the training policy and inference policy are not identical. An update that improves the training policy can inadvertently harm the inference policy, leading to instability or performance collapse.

What is the role of the "inference gap" in the acceptance test?

The inference gap measures the post-update discrepancy between the candidate training policy and its synchronized inference realization. A negative gap indicates that the inference engine is failing to realize the gains proposed by the trainer, signaling an unreliable update that should be rejected.

The Mirage of Optimizing Training Policies

Training‑inference mismatch makes improving the training policy insufficient for better inference.

Reinforcement learning for LLMs often updates a training‑side policy $π$ while generation uses a separate inference‑side policy $µ$. Even when parameters are synchronized, differences in precision and decoding cause $π$ and $µ$ to assign divergent probabilities to identical trajectories, creating a training‑inference mismatch that destabilizes learning. This mismatch leads to an objective misalignment: improving $π$ does not guarantee that $µ$—the policy deployed at inference—gets better.

The training engine’s policy $π$ and the inference engine’s policy $µ$ can assign different probabilities to the same rollout, even with identical model weights.

Why doesn’t improving the training policy $π$ automatically improve the inference policy $µ$?

Because $π$ and $µ$ are evaluated on different probability distributions; an update that raises $π$’s reward under the training‑side distribution may lower $µ$’s reward under the inference‑side distribution, so the improvement is not transferred.

**Figure 1.** Monotonic Inference Policy Update (MIPU) resolves the *Objective Misalignment* issue of LLM RL. Canonical LLM RL accepts synchronized updates by a training-side objective, which does not necessarily imply improvement of the inference policy. Here, $\pi$ and $\mu$ denote the training policy and inference policy respectively, $c$ is a tolerance parameter accounting for proxy noise. To address this mismatch, we propose a new principle as monotonic improvement on the inference-policy trajectory (the MIPI principle). MIPU realizes this principle with two steps: Step 1 optimizes Terms ②+③, while Step 2 estimates and validates Term ①, jointly covering all three terms in the MIPI decomposition.

Optimizing the training policy alone cannot ensure inference improvement because the two engines may evaluate trajectories differently.

Reinforcement Learning for LLMs

Defines the RL framing for LLMs and introduces the GRPO baseline.

Reinforcement learning treats LLM token generation as a sequential decision process, modeling each step as a state‑action pair in a Markov Decision Process.

At timestep $t$, the state $s_t$ consists of the prompt $q$ and the tokens generated so far, the action $a_t$ is the next token from vocabulary $V$, and the deterministic transition yields $s_{t+1}$.

The reward is zero for non‑terminal steps and equals one only if the final sequence $\tau$ is correct and well‑formatted, making the learning objective $J(\pi)=\mathbb{E}_{\tau\sim\pi}[R(\tau)]$.

Monotonic policy improvement seeks updates $\pi_{k+1}$ that guarantee $J(\pi_{k+1})-J(\pi_k)\ge0$, a property approximated in practice by surrogate objectives such as TRPO and PPO.

GRPO estimates advantages per group of trajectories, avoiding a learned value function while keeping updates stable.

The MIPI Principle

Introduces the Monotonic Inference Policy Improvement principle and its two‑step update mechanism.

MIPI forces each RL update to raise the expected return of the policy that will actually be used at inference time, rather than only improving a surrogate training policy.

How does MIPI differ from standard policy‑gradient updates that maximize $J(\pi_{k+1}) - J(\pi_k)$?

Standard updates improve the surrogate training policy $\pi$, which may diverge from the inference policy $\mu$. MIPI instead measures improvement against the current inference policy $\mu_k$, splitting the gap so that only updates that raise $J(\mu)$ are accepted, guaranteeing monotonic inference performance.

Pre‑update gap: $J(\pi_{k+1}) - J(\mu_k)=0.55-0.50=0.05$ (positive, so the candidate is promising).

Post‑update gap: $J(\mu_{k+1}) - J(\pi_{k+1})=0.60-0.55=0.05$ (non‑negative, so acceptance succeeds).

Overall inference improvement: $J(\mu_{k+1}) - J(\mu_k)=0.60-0.50=0.10$, which equals the sum of the two gaps.

The acceptance check ensures that even if the surrogate overestimates improvement, a negative post‑update gap would trigger a rollback, preserving monotonicity.

Acceptance and Rollback

We filter unreliable training updates by testing if the synchronized inference policy realizes the proposed performance gains.

Step 1 generates a candidate training policy $\pi_{k+1}$ that is theoretically beneficial, but it does not guarantee that the synchronized inference policy $\mu_{k+1}$ will actually realize those gains. Step 2 acts as a safety filter, checking for post-update consistency before committing to the change.

We treat the post-update inference gap as a risk signal: if the candidate policy $\pi_{k+1}$ assigns high probability to responses that perform poorly under the inference engine $\mu_{k+1}$, the update is likely unreliable and should be rejected.

This mechanism does not provide a formal guarantee of monotonic improvement, but it significantly reduces the risk of accumulating updates that look beneficial on the training side while failing to translate into better inference-time performance.

Implementation and Setup

We detail the update loop and experimental configuration for MIPU.

Algorithm 1 formalizes the two‑step MIPU loop: a sampler‑referenced proposal followed by a monotonic acceptance test.

Monotonic Inference Policy Update loop.

This step proposes a new trainer policy by using rollouts generated from the current inference policy, thereby grounding the update in the distribution that will actually be deployed.

How does this surrogate differ from the standard PPO clipped objective?

Standard PPO re‑weights advantages by a probability‑ratio clip, assuming the same policy generates data and is updated. Here we explicitly weight by $w_k = \pi_k / \mu_k$ because rollouts come from $\mu_k$, not $\pi_k$, so the correction targets the training‑inference mismatch rather than just policy‑ratio stability.

The update is accepted only if a validation‑derived inference‑gap metric shows the inference policy has improved, guaranteeing monotonic progress on the deployed policy.

Why not accept every update and rely on later training to correct mismatches?

Unconstrained updates can push the inference policy into regions where the KL gap widens, causing instability and degraded downstream performance. The acceptance test filters out harmful steps, preserving a monotonic improvement guarantee.

Training is performed on Qwen3‑1.7B and Qwen3‑4B with FP8‑quantized rollout, which deliberately inflates the training‑inference mismatch.

We use DAPO‑Math‑17 (5,759 examples) and DeepMath‑103K (1,491 examples) after filtering for non‑trivial solvability, providing diverse reward signals for RL.

Evaluation spans five math reasoning benchmarks (MATH‑500, AIME24, AMC23, Minerva, OlympiadBench) using pass@1; for AIME24 and AMC23 we report avg@16 to reduce variance.

We also report the inference‑training K3‑KL (Mismatch‑K3) computed as $\mathbb{E}[\exp(d_t) - d_t - 1]$ with $d_t = \log \pi(y_t|s_t) - \log \mu(y_t|s_t)$.

Baselines include the GRPO RL algorithm (referenced), two stabilization tricks (MIS, LR‑decay), and a TIS‑only variant that isolates Step 1.

Empirical Performance

MIPU delivers higher average accuracy and stable training under FP8‑quantized rollout.

The paper’s core premise is that optimizing a training policy that diverges from the inference policy harms performance; MIPU directly addresses this mismatch.

MIPU improves average Pass@1 accuracy by +2.97% over the baseline on Qwen3‑4B.

Table 1 shows baseline average 56.55% versus Ours 59.52%.

MIPU improves average Pass@1 accuracy by +3.11% over the baseline on Qwen3‑1.7B.

Table 1 reports baseline average 50.86% versus Ours 53.97%.

**Figure 2.** Performance of different methods under FP8-quantized rollout. Compared methods show unstable training dynamics and may suffer from sharp performance drops, while MIPU maintains a stable score trajectory.

Ablation Studies

Component ablations reveal how each step contributes to overall performance.

We evaluate the impact of each component of the proposed framework on the Qwen3‑4B model using FP8‑quantized rollouts. Step 1 builds sampler‑referenced candidate updates, while Step 2 decides whether to accept a synchronized candidate based on $T_{post}$. This ablation isolates their complementary contributions.

Adding Step 1 improves average pass@1 by +0.94 % over the baseline.

Table 2 shows 65.36 % for “+ Step 1” versus 64.42 % for “Baseline”.

Adding Step 2 alone reduces average pass@1 by –1.61 % relative to the baseline.

Table 2 reports 62.81 % for “+ Step 2” compared with 64.42 % for “Baseline”.

Combining Step 1 and Step 2 yields the highest average pass@1, +2.29 % over baseline.

Table 2 lists 66.71 % for the full method (“Ours”) versus 64.42 % for “Baseline”.

**Figure 3.** Training curves for ablation studies under FP8-quantized rollout. We show the training score, the inference-training K3-KL, $\hat{T}_{post}$ (i.e., inference gap) and the rollback rate computed over a 100-step moving window. Step 1 improves the candidate update direction, while Step 2 introduces inference-gap-aware acceptance to filter unreliable synchronized candidates. The full method obtains stronger performance with a more controlled inference-policy trajectory.

Overall, the ablations confirm that Step 1 and Step 2 address different failure modes: Step 1 supplies higher‑quality candidate updates, while Step 2 guards against inference‑policy collapse by rejecting risky candidates. Their combination delivers the best results.

Failure Mode Analysis

Step 2’s acceptance rule stabilizes training by using the post‑update gap signal.

Step 1 and Step 2 address distinct bottlenecks: Step 1 refines the update direction, while Step 2 decides whether the synchronized candidate becomes the next inference policy. Strong performance requires both a corrected update and an inference‑gap‑aware acceptance decision.

We first examine the post‑update gap proxy $\\tau_{post}$ as an inference‑side signal. Under FP8‑quantized rollout the smaller Qwen3‑1.7B model exhibits a larger training‑inference mismatch than Qwen3‑4B, and its $\\tau_{post}$ has a higher magnitude and stronger oscillations, indicating a volatile inference gap. Both models show a training‑dependent pattern: mismatch grows during training and $\\tau_{post}$ shifts from low to higher values.

To test whether Step 2’s benefit stems merely from rejecting updates, we compare it with a random‑rollback control that discards candidates without consulting $\\tau_{post}$. In a Step 2‑only run (no Step 1 correction) the rollback rate is about 70 % within the first 500 steps, so we set the random‑rollback probability to 70 %.

Despite rejecting fewer updates, Step 2 maintains a stable pass@1 trajectory, whereas random rollback collapses after a transient peak even though it is more conservative. This demonstrates that Step 2 is not a generic sparsification mechanism; its stability derives from conditioning acceptance on $\\tau_{post}$, which filters out candidates unlikely to improve the inference policy.

**Figure b.** Step 2 vs. random rollback.

Overall, inference‑gap‑aware acceptance lets useful candidates pass while discarding high‑risk ones, achieving both higher stability and comparable final performance.

Step 1 Implementation Analysis

Step 1 ablations dissect how each component affects performance and stability.

**Figure 5.** Step 1 implementation analysis under the Qwen3-4B FP8-quantized rollout. Comparison of PPO-IS, Vanilla-IS, and TIS in terms of performance, gradient norm, inference-training K3-KL, and clip ratio.

Step 1’s sampler‑referenced update replaces the naïve PPO‑IS ratio $\omega_{\text{PPO-IS}}$ with a factorized form that isolates the pre‑update mismatch $\pi_k/\mu_k$ from the current update $\pi_{\theta}/\pi_k$.

Instead of clipping the raw trainer‑to‑sampler ratio, we first correct for the existing mismatch between the current inference policy and the sampler, then apply clipping only to the fresh training update.

How does TIS differ from a standard importance‑sampling clip that simply bounds $\pi_{\theta}/\mu_k$?

Standard clipping mixes the old mismatch $\pi_k/\mu_k$ with the new update, so the bound can be triggered by stale policy drift. TIS first isolates the old mismatch, caps it at $C$, and then clips only the fresh update $\pi_{\theta}/\pi_k$, ensuring the trust‑region constraint reacts to the actual training step.

Tolerance Sensitivity Analysis

We evaluate TIS for Step 1 and study acceptance tolerance $c$ for Step 2, revealing trade‑offs.

TIS truncates the mismatch‑correction weight while preserving the decomposed update form. Compared with PPO‑IS it avoids overly aggressive clipping, and versus Vanilla‑IS it curbs variance amplification from large mismatch weights.

**Figure a.** Inference-training K3-KL and inference gap.

The acceptance tolerance $c$ governs Step 2: a candidate is kept if $T_{\text{post}} \ge -c$. Setting $c>0$ permits mild negative proxy values, $c=0$ enforces strict non‑regression, and $c<0$ imposes a stricter‑than‑zero rule.

**Figure 6.** Sensitivity to the acceptance tolerance $c$ under the Qwen3-4B FP8-quantized rollout in terms of inference-training K3-KL, training score, and $\widehat{T}_{\text{post}}$.

The dynamic tolerance used in MIPU starts with a larger $c$ to admit useful early updates, then gradually tightens as $T_{\text{post}}$ stabilizes, achieving a balance between rapid early learning and later‑stage protection against mismatch‑driven regressions.

Related Work

We expose why improving the training policy can hurt the deployed inference policy and reformulate the objective accordingly.

When the training policy $\pi$ differs from the inference policy µ, a step that raises the training return J($\pi$) does not guarantee a rise in the inference return J(µ), breaking the naïve assumption that better training automatically yields better deployment.

GRPO’s surrogate objective replaces the usual advantage term A_{$\pi_{k}$} with a group‑relative advantage \hat{A}_{\mu_k,i} estimated from inference rollouts, thereby inheriting the same distributional gap.

These two sources of mismatch expose an objective‑level misalignment: the surrogate still optimizes a training‑policy objective, yet the synchronized inference policy is what ultimately matters for deployment.

Sampler-Referenced Policy Update

Step 1 replaces the trainer‑to‑trainer ratio with a sampler‑referenced correction and clips only the fresh update.

Instead of clipping the trainer‑to‑trainer ratio, we first re‑weight the advantage by how the current policy compares directly to the inference‑policy sampler, then we clip only the fresh update factor.

Compute the truncated weight $\bar w_k=\min(3,2)=2$.

Clip the update ratio: $\operatorname{clip}(1.05,0.8,1.2)=1.05$ (unchanged because it lies inside the interval).

Form the surrogate contribution for this sample: $2 \times 1.05 \times 0.8 = 1.68$.

Average over all samples (here only one) gives $J_{S1}(\theta)=1.68$ for this minibatch.

Truncating $w_k$ prevents a large pre‑update mismatch from inflating the surrogate, while clipping only $r_i(\theta)$ keeps the update stable.

How does the sampler‑referenced variant differ from the standard GRPO surrogate that clips $\pi_\theta/\pi_k$?

GRPO clips the trainer‑to‑trainer ratio, which is centered on the old training policy $\pi_k$ even though samples come from the inference sampler $\mu_k$. The sampler‑referenced variant first re‑weights by the sampler‑to‑training mismatch $w_k$, then clips only the fresh update $r_i(\theta)$. This isolates the pre‑update mismatch from the actual policy change, avoiding over‑constraining the update when $w_k$ is large.

Compute the Step‑1 surrogate for a minibatch.

Hyperparameters

Key hyperparameter settings used in training and inference are summarized.

Table 3 collects the main hyperparameter choices across decoding, training, and infrastructure.

**Table 3.** Hyperparameter setups.

Implementation Details

Implementation details for the baseline and our method.

Baseline: we use vanilla GRPO with the same rollout, reward, and optimizer as competing methods. The implementation adds a dual‑clipped policy loss to the standard clipped objective. This loss caps both overly large and overly small policy ratios.

The loss limits the policy‑ratio for each token on both sides of a trust region, preventing extreme updates that could destabilize training.

Ours: for Step 1 we instantiate the sampler‑referenced update using token‑level truncated importance weights. For each generated token we compute a training‑to‑inference mismatch weight $w_k$, which scales the contribution of that token in the loss. The resulting objective is the Step 1 token‑level loss.

The weight $w_k$ measures how much a token’s probability under the training policy deviates from its probability under the inference policy, amplifying tokens where the two diverge.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers