Group Entropy-Controlled Policy Optimization

Guangran Cheng, Chengqi Lyu, Songyang Gao, Wenwei Zhang, Kai Chen

GEPO balances multi-task RL by using group entropy to adaptively shape advantage signals across heterogeneous tasks.

How can we prevent reinforcement learning for LLMs from collapsing when training on heterogeneous groups of responses with varying difficulty?

Reinforcement learning on heterogeneous task mixtures causes optimization imbalance, as standard group-based normalization fails to account for varying entropy regimes across different tasks. Group Entropy-Controlled Policy Optimization (GEPO) addresses this by using group entropy as a diagnostic signal to perform asymmetric advantage shaping, attenuating over-exploitative signals in low-entropy groups and preserving exploration in high-entropy groups. This approach achieves state-of-the-art performance across thirteen diverse benchmarks, delivering balanced improvements while preventing the catastrophic training collapses common in unconstrained group-based methods.

Paper Primer

GEPO functions as a lightweight, model-agnostic wrapper for group-based policy optimization: it computes group entropy from existing rollouts, then uses adaptive, smoothed thresholds to scale advantage signals up or down based on the task's current exploration state.

GEPO consistently outperforms existing entropy-controlled and group-based RL methods across diverse reasoning and instruction-following tasks.

On the Intern-S1-mini model, GEPO achieved an average benchmark score of 54.2, compared to 51.8 for the strongest baseline (KL-Cov). GEPO improved average performance by 2.4 points on Intern-S1-mini and 0.3 points on Qwen3.5-9B while maintaining superior training stability.

The method's core move is asymmetric advantage shaping: it applies a gentler attenuation to positive advantages in low-entropy groups to avoid length collapse, while more aggressively suppressing negative advantages in high-entropy groups to sustain exploration.

Why is global or token-level entropy regulation insufficient for multi-task LLM training?

Different tasks occupy distinct entropy regimes under the same policy; a global regulator treats all tasks identically, causing the optimization to be dominated by low-entropy tasks while high-entropy tasks receive noisy, ineffective gradient signals.

Does GEPO require task-specific tuning or manual annotations to handle different domains?

No; GEPO uses adaptive entropy thresholds derived from the empirical distribution of the current training batch, allowing it to automatically calibrate to different base models and shifting exploration states throughout training.

GEPO demonstrates that preserving differentiated exploration regimes across tasks is more effective than forcing heterogeneous tasks toward a uniform entropy pattern.

Researchers can now stabilize multi-task RL post-training by incorporating group-level entropy diagnostics, effectively decoupling task-specific exploration requirements from global optimization pressures.

Introduction and Motivation

We expose why uniform advantage normalization breaks in heterogeneous RL tasks and set the stage for GEPO.

Reinforcement learning for LLMs often applies entropy‑controlled methods, but when training on mixtures of diverse tasks the resulting groups exhibit distinct entropy regimes, making the uniform advantage normalization used by Group Relative Policy Optimization (GRPO) statistically biased and unstable. This heterogeneity causes advantage signals across prompt groups to be non‑comparable, undermining balanced exploration‑exploitation.

Uniformly normalizing advantages across heterogeneous response groups ignores their differing entropy levels, which leads to training instability.

Uniform normalization fails in heterogeneous RL, necessitating entropy‑aware shaping.

Foundations of Group RL

Introduce GRPO, the baseline RL framework that estimates advantages from grouped samples without a value function.

GRPO treats a batch of sampled responses as a single group, computes a shared advantage signal from their rewards, and updates the policy using that group‑level signal—so no separate value network is needed.

Mean $= (5+7+4)/3 = 5.33$, standard deviation $\approx 1.25$.

Normalized advantages: $A_1 = (5-5.33)/1.25 \approx -0.27$, $A_2 = (7-5.33)/1.25 \approx 1.34$, $A_3 = (4-5.33)/1.25 \approx -1.07$.

Assume each response has $T_i=2$ tokens and the raw ratios are $\rho_{1,1}=1.1$, $\rho_{1,2}=0.9$, $\rho_{2,1}=0.95$, $\rho_{2,2}=1.05$, $\rho_{3,1}=1.2$, $\rho_{3,2}=0.8$.

With $\epsilon=0.2$, the clipped ratios are $\operatorname{clip}(1.1)=1.1$, $\operatorname{clip}(0.9)=0.9$, etc.; the surrogate term for response 2 token 1 is $\min(0.95\times1.34,\,1.2\times1.34)=1.27$.

Aggregating all six token terms and averaging yields the final loss contribution for this batch.

Normalizing across the batch lets a high‑reward response pull the shared advantage up, while low‑reward samples are pulled down, stabilizing updates without a separate value network.

**Figure 1.** Comparison of entropy control methods at different granularities. (a): Policy-level methods calculate the entropy of policy over the current batch and unanimously apply it to regulate all responses; (b): Token-level methods calculate the covariance of tokens over the current batch and individually apply it to regulate each token; (c): GEPO calculate group entropy over the responses of each prompt and respectively regulate responses, enabling task-specific exploration-exploitation trade-off across heterogeneous tasks.

How does GRPO differ from standard PPO that relies on a learned value function?

Standard PPO estimates a per‑state value and computes advantages as $r - V$, requiring a separate network. GRPO skips the value network entirely by treating the batch of $K$ responses as a single “state” and normalizing their rewards, so the advantage is derived solely from the reward distribution of that batch.

Analyzing Group Entropy Dynamics

Group Entropy reveals why uniform advantage scaling breaks across domains.

Uniform advantage normalization assumes every response group explores similarly, but empirical runs show wildly different exploration levels. This mismatch drives instability in multi‑domain training.

Group Entropy measures how spread out a policy’s sampled responses are for a given prompt, acting as a per‑group probe of exploration versus over‑exploitation.

How does Group Entropy differ from the standard entropy of a single policy output?

Standard entropy treats a single sampled response as the whole distribution, yielding a noisy point estimate. Group Entropy aggregates multiple independent samples from the same prompt, averages their token‑wise log‑probabilities, and normalizes by length, producing a low‑variance, comparable signal across heterogeneous domains.

Compute log‑probabilities: $-\log 0.5 = 0.693$, $-\log 0.4 = 0.916$, $-\log 0.3 = 1.204$, $-\log 0.6 = 0.511$, $-\log 0.2 = 1.609$.

Sum per‑response: $S_1 = 0.693 + 0.916 = 1.609$; $S_2 = 1.204 + 0.511 + 1.609 = 3.324$.

Average over the two responses: $\hat{H}_g(x) = -(1/2)(1.609 + 3.324) = -2.466$ (negative sign omitted for clarity, the entropy estimate is $2.466$).

Average length $\bar{T} = (2 + 3)/2 = 2.5$; per‑token entropy $\mathcal{H}_g(x) = 2.466 / 2.5 \approx 0.987$.

Even with a tiny sample, the estimator yields a smooth entropy value that reflects both the confidence of the model (low probabilities increase entropy) and the differing response lengths, enabling fair comparison across domains.

**Figure.** (a) group entropy by domain.

**Figure (b).** Normalized advantage distributions by domain.

These empirical patterns expose the core flaw of standard group normalization: it treats all groups as if they share the same entropy, ignoring the systematic link between entropy, accuracy, and advantage skew that drives instability.

Theoretical Reward Distributions

We formalize how policy entropy distorts reward distributions and advantage scaling.

Uniform advantage normalization across heterogeneous response groups ignores the fact that policy entropy varies widely, causing unstable gradients and biased learning. Entropy‑dependent distortion is the key pain point that the paper’s method must address.

The reference reward CDF $F_{\nu}^{x}(t)$ describes how a uniformly random policy would rank rewards for a given prompt $x$, providing a neutral baseline against which the actual policy’s reward distribution can be compared.

How does the reference reward CDF differ from the policy‑weighted reward CDF?

The reference CDF assumes each action is equally likely (uniform $\nu_{x}$), while the policy‑weighted CDF $F_{\pi}^{x}$ weights actions by the learned policy $\pi_{x}$, so the latter reflects the actual sampling distribution of the model.

Compute $F_{\nu}^{x}(t)$: for $t=2$, two actions have reward $\le2$, so $F_{\nu}^{x}(2)=0.5$.

Assume a learned policy $\pi_{x}$ gives probabilities $(0.1,0.2,0.3,0.4)$. Then $F_{\pi}^{x}(2)=\pi_{x}\{a:\mu_{x}(a)\le2\}=0.1+0.2=0.3$.

Kolmogorov distance at $t=2$ is $|0.3-0.5|=0.2$.

Total variation $\operatorname{TV}(\pi_{x},\nu_{x})=\frac12\sum_{i}|0.1-0.25|+|0.2-0.25|+|0.3-0.25|+|0.4-0.25|=0.25$.

Entropy $H(x)=-\sum_{i}\pi_{x}(a_{i})\log\pi_{x}(a_{i})\approx1.85$ bits; $\log N_{x}= \log 4 =1.386$ (natural log) → $\log N_{x}-H(x)\approx -0.464$ (negative, so the bound is vacuous here, illustrating that high entropy yields a tighter bound).

The example shows how a low‑entropy policy (skewed probabilities) can produce a larger deviation between $F_{\pi}^{x}$ and the neutral baseline $F_{\nu}^{x}$, exactly the phenomenon captured by Proposition 2.1.

For any prompt $x$ with conditional policy entropy $H(x) = -\sum_{a}\pi_{x}(a)\log\pi_{x}(a)$ and response space size $N_{x}=|\mathcal{A}_{x}|$, the Kolmogorov distance between the policy‑weighted and reference reward CDFs satisfies $|F_{\pi}^{x}(t) - F_{\nu}^{x}(t)| \le \operatorname{TV}(\pi_{x},\nu_{x}) \le \sup_{t}\sqrt{\tfrac12(\log N_{x} - H(x))}$, where the second inequality holds when $\nu_{x}$ is uniform.

Assume $|\mu_{x}(a)| \le M$ and define $\sigma_{\pi}^{x}(a) = (\mu_{x}(a) - m_{\nu}^{x})$ with a constant $\sigma_{0}>0$ such that $\sigma_{\pi}^{x}(a) \ge \sigma_{0}$ for all $x,a$. Then there exists $C=C(M,\sigma_{0})>0$ with $\sqrt{\tfrac12}\,|a_{x}(a)| \le C\,\operatorname{TV}(\pi_{x},\nu_{x}) \le C(\log N_{x} - H(x))$, where $a_{x}(a) = \frac{\mu_{x}(a) - m_{\pi}^{x}(a) - Z_{\nu}^{x}}{\sigma_{\pi}^{x}}$, $Z_{\pi}^{x} = \frac{Z_{\pi}^{x}}{\sigma_{\nu}^{x}}$, and $\sigma_{\nu}^{x}=|Z_{\pi}^{x}|$.

Propositions 2.1–2.2 together create entropy‑dependent optimization pressures. Low‑entropy prompts can dominate the gradient signal because their advantages are most distorted, while high‑entropy prompts contribute weaker, noisier gradients, leading to a self‑reinforcing imbalance across groups.

The GEPO Algorithm

GEPO reshapes advantage updates by scaling them according to group entropy, dynamically balancing exploration across difficulty regimes.

Recall that uniform advantage normalization ignores the varying difficulty of response groups, causing instability. GEPO addresses this by conditioning the policy update on the entropy of each group.

We let the entropy of a response group dictate how strongly we scale its advantage signals, attenuating over‑confident or under‑confident updates.

Action 1 has $A_1>0$ and $\mathcal{H}_g=0.30$ > $\mathcal{H}_{\text{low}}$, so it is left unchanged: $\hat{A}_1=0.4$.

Action 2 has $A_2<0$ but $\mathcal{H}_g=0.30$ < $\mathcal{H}_{\text{high}}$, so it is left unchanged: $\hat{A}_2=-0.2$.

Action 3 has $A_3>0$ and $\mathcal{H}_g=0.30$ > $\mathcal{H}_{\text{low}}$, so it is left unchanged: $\hat{A}_3=0.1$.

If the same group had a lower entropy $\mathcal{H}_g=0.10$, then Action 1 would be scaled: $\hat{A}_1=0.5\times0.4=0.2$.

If a high‑entropy group had $\mathcal{H}_g=0.70$, then Action 2 would be scaled: $\hat{A}_2=0.2\times(-0.2)=-0.04$.

The piecewise scaling leaves advantages untouched when the entropy lies in the middle band, but aggressively attenuates signals that could destabilize learning at the extremes.

How does GEPO differ from the uniform advantage scaling used in standard group‑based RL?

Uniform scaling applies the same factor to every advantage regardless of the group’s difficulty, which can over‑penalize confident (low‑entropy) groups or under‑penalize uncertain (high‑entropy) groups. GEPO conditions the factor on entropy, using a larger $\alpha_{\text{low}}$ for low‑entropy positives and a smaller $\alpha_{\text{high}}$ for high‑entropy negatives, thereby preserving stability across regimes.

Collect a batch of prompts and compute the group entropy $\mathcal{H}_g(x)$ for each.

Calculate batch statistics $\mu_H(t)$ and $\sigma_H(t)$, then form provisional bounds $\hat{\mathcal{H}}_{\text{low}}(t)$ and $\hat{\mathcal{H}}_{\text{high}}(t)$ using $\beta_{\text{low}}$ and $\beta_{\text{high}}$.

Update the smoothed thresholds $\mathcal{H}_{\text{low}}(t+1)$ and $\mathcal{H}_{\text{high}}(t+1)$ with smoothing coefficient $\gamma$.

For each sampled action, apply the piecewise scaling in Equation (7) to obtain $\hat{A}_i$.

Feed the scaled advantages $\hat{A}_i$ into the underlying GRPO policy‑gradient update.

Proceed to the next training step.

Performance Evaluation

GEPO outperforms all baselines, delivering the highest average scores.

GEPO achieves the highest average score across benchmarks, beating the strongest baseline by 0.3 points.

Table 1 shows GEPO average 71.2 versus Clip‑Cov 70.9 and KL‑Cov 69.9.

**Table 1.** Main results on thirteen benchmarks across baselines and our methods

GEPO consistently outperforms GRPO across diverse benchmarks.

Training Dynamics and Stability

GEPO delivers stable validation and entropy trajectories across models.

GEPO avoids catastrophic validation collapse, keeping accuracy within 5% of its peak, whereas GRPO drops from ~85% to ~40% on AIME25.

Figure 4 shows GEPO’s validation curves rising smoothly, while GRPO’s curve plunges around step 170.

**Figure 3.** Typical validation curves during training process. For Intern-S1-mini and Qwen3.5-9B, GEPO improves smoothly throughout training, whereas baselines exhibit larger oscillations or late-stage performance collapse.

**Figure 4.** Policy entropy dynamics during training process of Intern-s1-mini. GEPO maintains a stable and bounded entropy trajectory, whereas the baselines exhibit runaway entropy growth or persistent oscillations.

**Figure 5.** Task entropy dynamics during training process of Intern-s1-mini. GEPO maintains stable, domain-specific entropy regimes, while the baselines either destabilize or homogenize them.

Ablation Analysis

We test each GEPO component by removing it and measuring the impact on benchmark scores.

To verify that every piece of GEPO is necessary, we run ablations on Intern‑S1‑mini and report scores for each benchmark.

Instead of applying the same advantage scaling to every group, the method uses a gentler factor for high‑entropy groups ($α_{\text{high}}$) and a stronger factor for low‑entropy groups ($α_{\text{low}}$), preserving exploration where it is needed.

How does Asymmetric Advantage Shaping differ from the usual symmetric advantage scaling?

Symmetric scaling uses a single scalar $α$ for every group, so high‑entropy groups receive the same reinforcement as low‑entropy ones. The asymmetric version computes two scalars, $α_{\text{high}}$ and $α_{\text{low}}$, based on group entropy; $α_{\text{high}}$ is smaller, preventing over‑reinforcement of already exploratory groups, while $α_{\text{low}}$ is larger, rescuing groups that would otherwise collapse.

Overall, the ablations confirm that each GEPO component contributes uniquely: high‑entropy control prevents uncontrolled entropy growth, low‑entropy control maintains diversity, and asymmetric advantage shaping balances the trade‑off.

Formal Advantage Analysis

Formal limits of per‑group normalization and its induced biases.

This appendix formalizes how per‑group normalization reshapes advantage signals and derives the resulting structural biases.

Normalizing advantages within each response group scales correct and incorrect outcomes differently, creating an asymmetry that depends on the group’s accuracy.

How does this differ from the standard advantage normalization used in vanilla policy gradient?

Standard normalization treats all samples identically, scaling by the global mean and variance. Here the scaling uses the group‑specific mean p and variance √{p(1−p)}, so groups with low accuracy get much larger magnitude advantages for the few correct samples and tiny magnitudes for the many incorrect ones.

When a group’s accuracy is low, the few correct responses receive large positive advantages while the many incorrect responses receive tiny negatives, concentrating the learning signal.

The asymmetry of the advantage distribution can be captured by its skewness, which becomes large for low‑accuracy groups.

Pooling groups with heterogeneous accuracies amplifies overall skewness because the skewness function is convex on (0, 0.5); Jensen’s inequality guarantees the pooled skewness exceeds the skewness of the average accuracy.

The difference between the policy‑weighted reward CDF and the uniform reference CDF cannot exceed the total‑variation distance between the policy and the reference distribution.

Standardizing advantages under the policy versus under a reference measure yields differences that are proportional to the policy’s total‑variation distance from the reference.

Formalizing advantage asymmetry under per‑group normalization reveals signal concentration and skewness that drive instability in group‑based RL.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers