Make Sparse Rewards Count: Density-Aware Reward Aggregation for Multi-Reward RL
Tong Zheng, Skylar Zhai, Zhan Cheng, TianMing Sha, Youling Huang, Shuo Zhou, Shaotong Qi, Jingcheng Liang, Xuwei Ding, Pengcheng Xu
DARA calibrates multi-reward RL by weighting rewards based on their active-group density to equalize advantage energy.
How can we prevent dominant rewards from overshadowing sparse rewards in multi-objective RL training?
In multi-reward reinforcement learning, different objectives often learn at uneven rates because some rewards provide useful comparisons more frequently than others. This creates a signal imbalance where "easier" rewards dominate the policy update while sparser, harder objectives are neglected. Density-Aware Reward Aggregation (DARA) corrects this by measuring the active-group density of each reward—the fraction of rollout groups where it provides a non-zero advantage—and applying an inverse-square-root weight to equalize the total "advantage energy" across all rewards. This calibration accelerates learning, reaching high format and length compliance in up to 26% and 65% fewer training steps, respectively, compared to standard reward-wise normalization.
Paper Primer
The core mechanism hinges on the observation that under reward-wise normalization, a reward's total influence on the policy gradient is proportional to its active-group density. DARA acts as a dynamic equalizer: it identifies rewards that are rarely active and boosts their weight to match the energy contribution of the most frequently active reward in the batch.
DARA significantly accelerates convergence on sparse-reward objectives.
In mathematical reasoning tasks, DARA reached near-saturated length compliance in up to 65% fewer training steps than GDPO.
DARA maintains competitive final performance compared to baseline methods.
After convergence, DARA-Asym achieves accuracy comparable to GRPO while enforcing constraints more effectively, as seen in Qwen3-4B-Instruct evaluations.
Why does simple reward-wise normalization fail to balance learning progress?
While reward-wise normalization ensures rewards contribute at comparable scales within a single group, it does not account for how often a reward is actually "active" across the entire batch. Rewards that are active in more groups accumulate more total "advantage energy," causing the model to prioritize them over sparser, harder-to-satisfy objectives.
How does DARA differ from simply increasing the weight of sparse rewards?
Fixed weighting is static and often leads to accuracy degradation by over-emphasizing a single objective. DARA computes weights dynamically per rollout batch based on observed activity, allowing the model to adapt to changing reward signals throughout the training process.
DARA is orthogonal to the underlying policy optimization objective (e.g., GRPO or PPO), meaning it can be dropped into existing multi-reward RL pipelines as a pre-aggregation calibration step without modifying the core training loop.
Researchers should treat active-group density as a primary diagnostic for multi-reward imbalance; DARA provides a robust, automated way to rebalance these signals without manual reward tuning.
Introduction and Motivation
We expose why multi-reward RL suffers from reward‑starvation and introduce DARA to rebalance reward signals.
Multi‑reward reinforcement learning frequently collapses onto dense rewards, leaving sparse objectives under‑optimized and causing reward‑starvation.
When dense rewards dominate, the gradient signal for sparse rewards becomes negligible, preventing those objectives from influencing the policy.
GDPO normalizes each reward dimension within rollout groups, preserving reward‑specific relative information before aggregation.
**Figure 1.** (a) DARA performs reward-wise group normalization, estimates active-group density, and applies density-based calibration before reward aggregation and batch-wise normalization. (b) Median format reward and active-group density, with shaded regions indicating the interquartile range. DARA reaches high format compliance in fewer training steps.
The DARA Mechanism
We describe DARA, a density‑aware weighting that equalizes reward energy across objectives.
Uneven learning across multiple reward signals stems from dense rewards monopolizing the gradient signal, leaving sparse rewards under‑driven. We therefore design a mechanism that rebalances the contribution of each reward based on how often it is active within rollout groups.
It measures the fraction of rollout groups in which a particular reward actually varies, i.e., provides a learning signal.
Group 1 average advantage $= (1+0+1)/3 = 0.67 > \tau$, so $I_{1,k}=1$.
Group 2 average advantage $= 0 < \tau$, so $I_{2,k}=0$.
Density $\pi_{k}= (1+0)/2 = 0.5$, meaning the reward is active in half of the groups.
Even a binary reward can be highly informative if it flips within a group; the density captures this nuance.
How does Active-Group Density differ from simply counting how many times a reward appears as “1” in the batch?
Counting occurrences ignores whether the reward varies within a rollout group. Density requires a non‑zero contrast inside a group, so a reward that is always “1” across a group contributes nothing to learning.
It aggregates the squared advantage coefficients of a reward across the whole batch, quantifying the total “push” that reward can exert on the policy.
Energy for reward 1: $E_{1}=B\pi_{1}(G-1)=2\times0.5\times2=2$.
Energy for reward 2: $E_{2}=2\times0.25\times2=1$.
Reward 1 supplies twice the total advantage magnitude of reward 2.
Even though both rewards appear equally often, the one that varies more across groups accumulates higher energy.
Why square the advantages instead of summing their absolute values?
Squaring emphasizes larger deviations while preserving sign information in later calibration steps; it also yields a clean analytical link to active‑group density under GDPO.
We re‑weight each reward by the inverse of its active‑group density so that all rewards contribute comparable energy to the policy update.
Reference density $\pi_{\text{ref}}=\max(0.8,0.2)=0.8$.
Weight for reward 1: $w_{1}= \min(4,0.8/0.8)=1$.
Weight for reward 2: $w_{2}= \min(4,0.8/0.2)=\min(4,4)=4$ (capped at $w_{\max}$).
Symmetric calibrated advantage for reward 2: $\tilde{A}^{2,\text{sym}}=4\,A^{2}$, boosting its signal fourfold.
Asymmetric calibrated advantage for reward 2: $\tilde{A}^{2,\text{asym}}=4\,A^{2}+(4-1)[A^{2}]^{+}$, further emphasizing positive contributions.
By inflating the sparse reward’s weight, DARA equalizes its total energy with the dense reward, preventing the latter from drowning out the former.
How does DARA differ from simply scaling each reward by the inverse of its occurrence frequency?
Frequency scaling ignores the group‑level contrast captured by active‑group density and does not cap amplification, which can lead to unstable gradients. DARA uses $\pi_{k}$, caps the weight, and distinguishes positive from negative advantages.
Compute per‑reward advantages $A_{i,j}^{k}$ for every rollout.
Determine active‑group indicators $I_{i,k}$ and batch densities $\pi_{k}$.
Identify the reference density $\pi_{\text{ref}}=\max_{k}\pi_{k}$.
Compute density‑based weights $w_{k}$ with cap $w_{\max}$.
Apply the chosen variant (sym or asym) to obtain calibrated advantages $\tilde{A}_{i,j}^{k,v}$.
Aggregate calibrated advantages across rewards to form $S_{i,j}^{v}$.
Normalize $S_{i,j}^{v}$ across the batch to produce $\hat{S}_{i,j}^{v}$.
Use $\hat{S}_{i,j}^{v}$ as the advantage input to the clipped policy‑optimization objective.
Tool-Calling Performance
Tool-Calling experiments quantify DARA’s gains in accuracy and format compliance.
ToolRL trains a language model to pick an external tool and to generate the tool’s arguments given a user query and tool descriptions.
How does ToolRL differ from a standard language‑model fine‑tuning task?
Standard fine‑tuning predicts free‑form text, while ToolRL requires the model to output a rigid, two‑part structure (tool name + arguments). This forces the model to learn a discrete selection step and a constrained generation step, which introduces a sparse reward component that ordinary language modeling does not have.
**Table 1.** BFCL-v4 performance at step 100. Acc. denotes tool-calling accuracy and Format denotes output-format compliance. Average is the mean over Live, Non-Live, and Multi-Turn.
**Figure 2.** Checkpoint-wise BFCL-v4 evaluation of Qwen2.5-1.5B-Instruct. Left: Average Accuracy; right: Average Format, each averaged over Live, Non-Live, and Multi-Turn.
**Figure 3.** Format active-group density across rollout-group sizes. Curves show the median and shaded regions the interquartile range. Markers on the dashed line show when format reward first reaches 0.8; open markers indicate that it does not reach 0.8 within 100 steps.
DARA improves tool‑calling accuracy and format compliance.
Mathematical Reasoning Results
Dynamic reward weighting lets DARA hit full length compliance far faster than baselines.
DARA‑Sym attains perfect (100 %) length‑compliance by step 100, outpacing every baseline.
Table 2 shows DARA‑Sym reaching 100 % length reward while GRPO and GDPO lag behind.
**Figure 4.** Training reward dynamics for mathematical reasoning on DeepSeek-R1-1.5B. Left: correctness reward; right: length reward. Static denotes the Static-Sym control in Table 4(a).
**Figure 5.** First stable crossing step for increasingly strict training-side length-compliance thresholds on Qwen3-4B-Instruct (left) and DeepSeek-R1-7B (right). Lower values indicate faster acquisition.
**Figure 6.** Checkpoint-wise held-out mathematical reasoning evaluation on DeepSeek-R1-7B. Left: macro pass@1 accuracy; right: percentage of responses exceeding 4,000 tokens.
**Table 2.** BFCL-v4 performance at step 100 of Qwen2.5-1.5B-Instruct trained with correctness, format, and length rewards. Len. denotes compliance; the other columns follow Table 1.
**Table 4.** Left: dynamic versus fixed reward weighting on DeepSeek-R1-1.5B. Right: comparison between Qwen3-4B Instruct and Thinking; underlined values mark DARA-Asym.
**Table 1.** (a) Fixed-weight ablation and (b) Effect of thinking mode.
Density-Energy Analysis
Brief derivations confirming the density–energy links and their impact on gradient behavior.
Multi‑reward RL often stalls because dense rewards drown out sparse ones; DARA mitigates this by re‑weighting rewards according to the active‑group density.
For any active rollout group $i\) with size $G$, the total advantage energy contributed by reward $k$ is $$E_{k}$=B\pi_{k}(G-1)$, i.e. linear in the active-group density $\pi_{k}$.
Compute the sample variance of reward $k$ for group $i$ (Equation 9).
Show that the sum of squared advantages equals $G-1$ (Equation 10).
Count active groups.
Aggregate energy over all active groups.
The uncapped coefficient $\alpha^{\star}_{k}= \dfrac{\pi_{\text{ref}}}{\pi_{k}}$ equalizes the calibrated advantage energy of reward $k$ to the reference energy $E_{\text{ref}}=B\pi_{\text{ref}}(G-1)$.
Express calibrated energy as a function of $\alpha_{k}$.
Insert the density–energy relation $E_{k}=B\pi_{k}(G-1)$.
Set calibrated energy equal to the reference energy $E_{\text{ref}}=B\pi_{\text{ref}}(G-1)$.
Solve for $\alpha_{k}$.
For any rewards $k$ and $\ell$, the conditional inner product of their policy‑gradient contributions satisfies $\mathbb{E}\!\langle g_{k},g_{\ell}\rangle\mid F = \langle h_{k},h_{\ell}\rangle + \frac{c}{N^{2}}\langle A_{k},A_{\ell}\rangle + R_{k\ell}$, with $|R_{k\ell}|\le \frac{\delta c}{N^{2}}E_{k}E_{\ell}$ and $c>0$, $0\le\delta<1$.
Decompose $z(i,j)$ into mean and fluctuation.
Insert the decomposition into $g_{k}$.
Take the conditional inner product of $g_{k}$ and $g_{\ell}$.
Approximate $C$ by $cP$ and bound the deviation.
Bound $R_{k\ell}$ using $\|A_{k}\|^{2}=E_{k}$.
If the $G$ outcomes for prompt $i$ are conditionally independent with success probability $\pi_{k,i}$, then $\mathbb{E}[E_{k}\mid x_{1},\dots,x_{B}] = (G-1)\frac{1}{B}\sum_{i=1}^{B} q_{G}(\pi_{k,i})$, where $q_{G}(p)=1-p^{G}-(1-p)^{G}$.
Define the activity indicator for a single rollout.
Compute its expectation under independence.
Sum over $B$ prompts and multiply by $(G-1)$.
For group size $G\ge2$ dividing $N$, the expected active‑group density $\bar{q}_{k,G}$ and per‑response advantage energy satisfy $\mathbb{E}[\pi_{k}] = \bar{q}_{k,G}$ and $\frac{\mathbb{E}[E_{k}]}{N}= 1-\frac{1}{G}\,\bar{q}_{k,G}$, both non‑decreasing in $G$ and strictly increasing when $0<\pi_{k}(X)<1$ with positive probability.
Express $\mathbb{E}[\pi_{k}]$ as the average of $q_{G}(\pi_{k,i})$.
Derive the per‑response energy.
Show monotonicity of $q_{G}(p)$ in $G$.
Conclude the monotonicity of $\bar{q}_{k,G}$ and the derived quantities.
For the asymmetric transformed advantages $A^{\text{asym}}_{k}= A_{k} + (w_{k}-1)[A_{k}]^{+}$ with $w_{k}\ge1$, the resulting energy satisfies $E_{k}\le\|A^{\text{asym}}_{k}\|^{2}\le E_{k}+w_{k}^{2}E_{k}$, i.e. at most a factor $w_{k}^{2}$ increase over the baseline.
Write the squared norm of the asymmetric advantage vector.
Bound the cross term by Cauchy–Schwarz.
Replace $\|[A_{k}]^{+}\|^{2}$ with $E^{+}\le E_{k}$ and $\|A_{k}\|^{2}=E_{k}$.
Combine lower and upper bounds.
Advantage Computation Details
Defines the advantage calculations for multi‑reward RL and details the DARA algorithm.
Section B.1 introduces the core advantage estimators used across multi‑reward methods. Understanding these formulas is required before the DARA-specific steps are presented.
Section B.2 provides the concrete DARA algorithm that turns the estimators above into per‑token advantages for a rollout batch. The pseudocode follows the same notation introduced earlier.
DARA Advantage Computation for One Rollout Batch
Implementation and Configuration
We detail the training setup, evaluation protocol, and configuration tables for the tool‑calling experiments.
All methods share the same vLLM‑based training stack derived from the released GDPO codebase, ensuring identical model, rollout engine, optimizer settings, and training budget across experiments.
**Table 6.** Training configuration for the tool-calling experiments.
The actor’s parameters, gradients, and optimizer state stay on‑device, whereas reference parameters are offloaded; the 3 B variant also enables expandable CUDA allocator segments for larger memory footprints.
Evaluation follows the BFCL‑v4 protocol: each final checkpoint is run once per case using the same ToolRL prompting and response‑parsing pipeline, and scores for Live, Non‑Live, and Multi‑Turn subsets are averaged equally.
The table outlines the evaluation settings used for the model, including the evaluator version (BFCL-v4), subsets (Live, Non-Live, and Multi-Turn), and various inference parameters such as temperature (0.6), top-p (0.95), maximum new tokens (8,192), context length (32,768), inference seed (0), top-k (-1), and the inference engine (vLLM 0.11.0).
**Table 8.** Batch configuration for the rollout-group analysis.
**Table 9.** Training configuration for the mathematical-reasoning experiments.