Sharpening Tax in Post-Training
Changdae Oh, Qi Zeng, Qi Qi, Andrey Zhmoginov, Deren Lei, Yun He, Hoang Phan, Hangoo Kang, Azalia Mirhoseini, Sharon Li
Post-training RL sharpens LLM policies at the cost of solution coverage, a trade-off quantified by the Sharpening Tax.
Does RL post-training fundamentally improve LLM reasoning, or does it merely "sharpen" existing capabilities at the expense of solution coverage?
Reinforcement learning (RL) post-training is widely assumed to unlock new agentic capabilities, but it often forces models to trade broad solution coverage for higher single-shot accuracy. The authors show that base models, when paired with a simple inference harness, can match or exceed the coverage of post-trained models given enough test-time compute. They introduce "Sharpening Tax" to quantify this loss and "Posterior-Tempered Group Sampling" (PTGS) to mitigate it. Across 42 model-benchmark combinations, post-training consistently bimodalizes success rates, concentrating mass at the extremes of "always solved" or "never solved" and reducing the value of repeated sampling.
Paper Primer
The "sharpening hypothesis" suggests that RL post-training merely amplifies a few high-reward behaviors already present in the base model. The authors test this in agentic tasks—which require multi-turn tool use and environment interaction—and find that post-training collapses the policy's success distribution, making it more consistent but less capable of recovering from initial failures through repeated sampling.
To diagnose this, the authors define Sharpening Tax: a metric measuring the expected number of failed attempts before a success, normalized by the single-shot failure rate. They propose PTGS, a Bayesian sampler that adaptively adjusts temperature per prompt based on estimated difficulty, heating hard prompts to increase success probability and cooling mastered ones to ensure consistency.
Post-training charges a pervasive Sharpening Tax across modern LLM families.
In 36 of 42 model-benchmark combinations, the calibrated tax is positive at a budget of 128 rollouts, indicating that post-trained models lose the ability to convert failures into successes via test-time compute. The tax is predictable from as few as 8 rollouts, with a Spearman correlation of $\rho = 0.85$ between early estimates and full-budget outcomes.
PTGS improves both accuracy and coverage simultaneously.
By adaptively tempering temperatures during RL training, PTGS increases the frequency of mixed-success rollout groups, providing a richer learning signal for PPO and GRPO compared to fixed-temperature baselines.
Why do base models perform so well in agentic tasks when they are supposedly less capable than post-trained ones?
Base models possess a higher performance ceiling (coverage) but lack the instruction-following and tool-calling interfaces. When equipped with a lightweight, model-agnostic harness, they can access these capabilities and leverage their broader behavior distribution to solve tasks that post-trained models have "sharpened" away.
Is the Sharpening Tax always a negative outcome?
Not necessarily. For smaller models or scenarios with strictly limited test-time compute, sharpening is beneficial because it raises the performance floor (pass@1). The tax becomes a liability primarily for larger models where the goal is to maximize coverage through extensive test-time sampling.
Researchers should evaluate post-trained models not just by single-shot accuracy, but by their Sharpening Tax. Future RL pipelines should incorporate adaptive sampling like PTGS to preserve the base model's latent potential while improving reliability.
The Sharpening Tax in Post-Training
We expose the trade‑off between post‑training accuracy gains and coverage loss, and introduce Sharpening Tax and PTGS.
Post‑training via reinforcement learning (RL) appears to sharpen the base model’s policy, boosting single‑shot accuracy but collapsing the distribution of successful rollouts, which reduces coverage across many attempts. This trade‑off has been documented on math and coding benchmarks, yet it is unclear whether it persists for agentic tasks that require multi‑turn tool use and long‑horizon planning.
RL post‑training fine‑tunes a pre‑trained LLM by rewarding trajectories that achieve a goal, effectively sharpening the policy toward high‑reward behaviors.
Sharpening Tax measures how much post‑training reduces the model’s ability to scale performance with additional rollouts, i.e., the loss in test‑time scalability.
PTGS adapts the sampling temperature for each prompt based on an estimate of its difficulty, smoothing the policy for hard tasks and sharpening it for easy ones.
**Figure 1. Project overview.** We investigate whether post-training fundamentally broadens the agentic reasoning capacity of its base model. (a) Our observations suggest that post-training sharpens the policy to improve sampling efficiency and consistency while compromising coverage. To quantify this systematically, (b) we present **Sharpening Tax**, measuring the difference in test-time scalability between base and post-trained LLMs. To lower the tax, (c) we then propose **Posterior-Tempered Group Sampling (PTGS)**, dynamically adjusting the sampling temperature based on task difficulty.
Post‑training improves consistency but sacrifices coverage.
Measuring Reasoning Boundaries
We define the pass@k family of metrics that quantify accuracy, coverage, and consistency of LLM rollouts.
Evaluating LLM reasoning boundaries relies on the pass@k family of metrics, which capture how often a policy succeeds within a fixed budget of $k$ independent rollouts per task.
Pass@K measures the probability that at least one of $k$ attempts succeeds, turning a single‑shot success rate into a multi‑shot coverage estimate.
Three complementary metrics are derived from pass@k: accuracy ($\text{pass@1}$) quantifies the single‑shot success rate; coverage ($\text{pass@k}$) quantifies the chance of finding at least one solution within $k$ attempts; consistency ($\text{pass}^{k}$) quantifies the probability that *all* $k$ attempts succeed.
We evaluate on three agentic benchmarks—BFCL v4 (multi‑turn tool‑calling), WebShop (interactive web navigation), and ACEBench (complex action sequences)—which verify intermediate actions rather than only final answers, avoiding inflated pass@k scores from lucky final hits.
Our experiments use open‑source checkpoint pairs (e.g., gemma‑4‑31B vs. gemma‑4‑31B‑it) covering 14 backbones from Gemma‑4, Ministral‑3, Qwen2.5, and Qwen3.5, ranging from 3 B to 35 B effective parameters. All post‑trained models have thinking mode disabled unless otherwise noted.
**Figure 10** Ablation on thinking mode. Coverage (pass@k) and consistency (pass^k) curves of the base and post-trained models of gemma-4-31B (left) and Qwen3.5-35B-A3B (middle) on BFCLv4 MT, with the thinking mode of the post-trained model turned off (our default setup) and on, and the corresponding `Tax_S`(k) (right), as a function of the rollout budget k per task. Thinking mode improves the accuracy and coverage of the post-trained models, but it still doesn’t change the bold conclusion—base models still catch up at large k, and the tax remains non-negative.
The Accuracy-Coverage Tension
Base models with a harness eventually outpace post‑trained models in coverage, especially at larger scales.
Base models with a harness surpass post‑trained models in pass@128 on WebShop.
Base reaches 85 % pass@128 whereas post‑trained attains 56 %.
Post‑training raises single‑sample success (high pass@1) but flattens the growth of success probability as more rollouts are allowed, whereas base models start lower but improve faster with additional rollouts.
How does this tension differ from the classic accuracy‑coverage trade‑off in static classification?
In static classification the trade‑off stems from a fixed confidence threshold, whereas here the tension arises from the rollout‑dependent success distribution shaped by post‑training sharpening; the distribution changes with K, not with a static threshold.
**Figure 2.** Base models catch up to their post-trained counterparts on agentic tasks. We draw pass@K curves (up to 128 rollouts per task) for four different models on three benchmarks. Post-trained models (orange) dominate at small K, but pre-trained base models equipped with a light harness (teal) scale more steeply and cross over in most settings.
**Figure 3.** Larger models pull the crossover points earlier. Pass@K test-time scaling curves of harness-equipped gemma-4 base models (teal) vs. their post-trained counterparts (orange) with model size increasing from left to right. Overall, the crossover budget at which the base model overtakes post-trained model shrinks as model scale grows.
**Figure 5** Post-training trades coverage for consistency. Mean pass@K and pass^K over 12 model-benchmark pairs.
**Table 1.** Harness ablation study of base models. Average pass@1 and pass@32 of eight base models from the Gemma-4, Ministral-3, Qwen2.5, and Qwen3.5 families with (w/) and without (w/o) harness. Table 4 provides per-model results.
Base models often retain higher potential coverage than post‑trained models.
Quantifying the Sharpening Tax
Sharpening Tax quantifies post‑training coverage loss in a single number.
Sharpening Tax estimated from 8 rollouts predicts its value at 32 rollouts with high rank correlation.
Spearman correlation $\rho$ = 0.85 between `Tax_S`(8) and `Tax_S`(32) across 42 model‑benchmark pairs.
Posterior-Tempered Group Sampling
Balancing sampling efficiency with coverage by adapting temperature per task.
RL post‑training sharpens the policy, boosting single‑shot accuracy but collapsing the diversity of solutions that the base model could explore. This section asks whether we can keep the efficiency gains while restoring coverage.
PTGS adapts the rollout temperature for each prompt by maintaining a Bayesian estimate of how hard that prompt is, then tempering the temperature so that hard prompts are sampled more exploratively and easy prompts more conservatively.
How does PTGS differ from simply raising the temperature for all prompts?
Uniformly increasing temperature boosts exploration but also harms the high‑accuracy regime, because easy prompts receive unnecessary noise. PTGS tailors the temperature per prompt: hard prompts get heated (high $T_x$) while easy prompts are cooled (low $T_x$), preserving the efficiency of low‑temperature sampling where it already works.
Step 1: Compute posterior parameters $a_x=2\tilde p+\tilde s_x=1+0=1$, $b_x=2(1-\tilde p)+\tilde f_x=1+0=1$.
Step 2: Sample $\hat p_x\sim Beta(1,1)$, which is uniform on $[0,1]$; suppose we draw $\hat p_x=0.3$.
Step 3: Apply the tempering function $h$ (decreasing) – assume $h(0.3)=2.0$ – and set $T_x=\tau h(\hat p_x)$ with $\tau=1.5$, giving $T_x=3.0$.
Step 4: Generate 4 rollouts at temperature $3.0$, observe $s_x=2$ successes.
Step 5: Update discounted counts: $\tilde s_x\leftarrow\gamma\tilde s_x+s_x=0.9\cdot0+2=2$, $\tilde f_x\leftarrow\gamma\tilde f_x+(n-s_x)=0.9\cdot0+2=2$.
Step 6: Next iteration recomputes $a_x=1+2=3$, $b_x=1+2=3$, narrowing the Beta distribution.
As the posterior concentrates, $h(\hat p_x)$ shrinks, automatically cooling the temperature for this prompt while still allowing occasional heating if future successes are sparse.
Algorithm 1 PTGS within an RL training loop
**Figure 8.** Illustration of posterior-tempered group sampling. Given a policy, we model its success rate for each task (prompt) $x$ as a dynamic difficulty estimate to adaptively determine rollout sampling temperature in RL training pipeline. It encourages exploratory behavior for hard prompts while inducing conservative behavior for easy ones.
**Figure 9** Training dynamics on FrozenLake. Validation success of intermediate checkpoints (left) and average entropy of the output token distribution during training for PPO and PPO with PTGS.
Theoretical Analysis
We quantify the Sharpening Tax, show its proportional impact, and prove PTGS reduces it via higher group success.
Post‑training sharpening boosts single‑shot accuracy but pushes tasks toward deterministic extremes, inflating a hidden cost on retries. This hidden cost is the Sharpening Tax we now formalize.
Let $H$ be the index of the first successful rollout for a randomly drawn task. Then, for a rollout budget $K$, $$A(K) = \mathbb{E}\big[(H - 1)\mathbf{1}\{H \le K\}\big],$$ $$S(K) = \mathbb{E}\!\left[\frac{H-1}{K-1}\mathbf{1}\{H \le K\}\right],\qquad S(K)\in[0,1].$$
If post‑training sharpens each task with probability $\lambda\in[0,1]$, replacing its base success probability $p_x$ by $0$ or $1$, then for any budget $K\ge2$, $$\text{Tax}A(K)=\lambda\,A_{\text{Base}}(K)\ge0,$$ $$ \frac{S_{\text{Post}}(K)}{S_{\text{Base}}(K)}= \frac{(1-\lambda)\,\varepsilon}{(1-\lambda)\,\varepsilon+\lambda_0},\qquad \varepsilon=1-\text{pass}@1_{\text{Base}}. $$
Compute $A_{\text{Base}}(K)=\sum_{k=1}^{K-1}[(1-p)^k-(1-p)^K]= (0.6)+(0.36)-0.216=0.744$.
Tax $A(K)=\lambda\,A_{\text{Base}}(K)=0.5\times0.744=0.372$.
First‑attempt failure rate $\varepsilon=1-\text{pass}@1_{\text{Base}}=1-(1-(1-p))=p=0.4$.
Assume $\lambda_0=0.2$ (20% of sharpened tasks go to $p=0$). Then $S_{\text{Post}}/S_{\text{Base}}=\frac{0.5\times0.4}{0.5\times0.4+0.2}=0.5$.
The tax removes exactly half of the base model’s expected retries, illustrating how sharpening directly penalizes the benefit of additional rollouts.
For a prompt whose success probability rises from $p$ to $p'$ under PTGS, and a group of $n\ge2$ independent rollouts, $$I_n(q)=1-(1-q)^n,\qquad J_n(q)=1-q^n-(1-q)^n.$$ If $p'>p$, then $I_n(p')>I_n(p)$ and $J_n(p')>J_n(p)$ for all $0\le p<p'\le1$.
**Figure 40.** PTGS reduces the proportion of zero-success rollout groups. Percentage of the rollout groups at each training step in which none of the 16 rollouts succeeds, i.e., groups that carry no success signal for the update, for PPO and PPO with PTGS. The proportion is measured before the reward-variance filter and averaged over Sokoban and FrozenLake. Thin lines show the mean over the ten runs of each method (five per environment) with a 9-step moving average as the thick line.
Together, these results explain why sharpening incurs a proportional tax and how PTGS mitigates it by making hard prompts more learnable, ultimately improving both single‑shot accuracy and coverage.
Related Work
We situate our work among prior studies on RL post‑training, diversity preservation, and test‑time scaling.
One line of work argues that RL‑tuned models lose pass@K at large K, suggesting post‑training sharpens the policy around behaviors already present in the base model. Another line claims the reasoning boundary can genuinely expand through prolonged RL training or grokking. We contribute a quantitative diagnosis, the Sharpening Tax, that turns visual scaling trends into a measurable quantity.
Preserving diversity in post‑training has motivated exploration bonuses, entropy‑collapse mitigation, and pass@K‑aware objectives. Sampling‑side approaches adjust rollout temperature, and our PTGS distinguishes itself by using a Beta‑Binomial posterior over success rates to guide difficulty‑aware temperature adaptation without altering the underlying RL algorithm.
Repeated sampling is a well‑studied axis of test‑time compute, with performance growing smoothly with the number of rollouts. Prior work treats it as a means to boost accuracy, but we repurpose the scaling curve itself as a diagnostic tool to reveal how post‑training reshapes the policy.
Aggregate Results
Additional empirical analysis of post‑training effects across model scales.
Post‑training improves coverage for the smallest backbones while reducing it for the largest backbones, with the base model overtaking the post‑trained model at around $k\approx22$ rollouts for the largest set.
Figure 12 shows the crossover point and Table 6 quantifies the task‑category shifts.
The contrast between scales persists across families: Figures 13–21 illustrate that larger models polarize per‑task success rates, whereas smaller models mainly gain new solvable tasks.
**Figure 12** Aggregate coverage (pass@k) and consistency (pass^k) curves of the base and post-trained models for the largest (left) and smallest (right) backbones, averaged over 12 model-benchmark combinations (one backbone per family on each of the three benchmarks). The consistency of the base model drops to nearly zero as k grows, but that of the post-trained model stays much higher at both scales. For the largest backbones, more samples let the base model overtake the post-trained model in coverage, but not in consistency.
**Figure 13** Aggregate per-task success-rate distributions of the base and post-trained models for the largest (left) and smallest (right) backbones, averaged over 12 model-benchmark combinations (one backbone per family on each of the three benchmarks). For the largest backbones, post-training increases both the never-solved share and the always-solved share (the latter from nearly 0% to 37%), i.e., it loses some solvable tasks and fully solves others. For the smallest backbones, post-training instead reduces the never-solved share by repairing the base model.
**Figure 14** Per-task success-rate distributions of the Gemma-4 base and post-trained models (empirical success rate over 128 rollouts per task) for every backbone and benchmark. Post-training moves most of the intermediate mass to the two extremes at every model size and on every benchmark. Among the four families, this polarization is the strongest for Gemma-4, which also has the largest mean tax.
**Figure 15** Per-task success-rate distributions of the Qwen2.5 base and post-trained models (empirical success rate over 128 rollouts per task) for every backbone and benchmark. Post-training moves the intermediate mass to the two extremes, as in the other families.
**Figure 16.** Per-task success-rate distributions of the Qwen3.5 base and post-trained models (empirical success rate over 128 rollouts per task) for every backbone and benchmark. Qwen3.5 shows the softest sharpening among the four families.
**Figure 17** Per-task success-rate distributions of the Ministral-3 base and post-trained models (empirical success rate over 128 rollouts per task) for every backbone and benchmark. The polarization grows with model scale. For the 3B backbone, the base model rarely succeeds on BFCL and WebShop and post-training largely improves it, while the 8B and 14B backbones show the shift of mass to the two extremes as we observed in other backbones.
**Figure 18** Coverage (pass@k) and consistency (pass^k) curves of the Gemma-4 base and post-trained models for every backbone and benchmark. The two curves of the post-trained model stay close together, as expected from a bimodalized policy. In contrast, the consistency of the base model drops toward zero while its coverage keeps rising. The base model overtakes the post-trained model in coverage at a smaller budget for larger backbones.
**Figure 19.** Coverage (pass@k) and consistency (pass^k) curves of the Qwen2.5 base and post-trained models for every backbone and benchmark. The consistency of the base model drops toward zero while its coverage keeps rising, and the two curves of the post-trained model stay much closer together.
**Figure 20.** Coverage (pass@k) and consistency (pass^k) curves of the Qwen3.5 base and post-trained models for every backbone and benchmark. Since Qwen3.5 has the softest sharpening among the four families, the consistency curves of its post-trained models decay slightly instead of staying flat, and the coverage gaps between the base and post-trained models at k = 128 are the smallest among the four families.
**Figure 21.** Coverage (pass@k) and consistency (pass^k) curves of the Ministral-3 base and post-trained models for every backbone and benchmark. The consistency of the base model drops toward zero while its coverage keeps rising, and the two curves of the post-trained model stay much closer together.
**Figure 22** Coverage lost versus sampling efficiency gained by post-training at $K=8$ (left) and $K=128$ (right). The $y$-axis is the coverage lost, $\text{pass}@K_{\text{base}} - \text{pass}@K_{\text{RL}}$, and the $x$-axis is the efficiency gained, $\text{avg}@K_{\text{RL}} - \text{avg}@K_{\text{base}}$, where $\text{avg}@K$ is the mean success rate over $K$ rollouts. Each point is one model-benchmark combination, colored by family, and the marker size grows with the total number of parameters. At $K=8$, most points lie below zero, i.e., post-training is beneficial in both coverage and efficiency when compute is scarce. At $K=128$, many points, mostly from the larger backbones, cross above zero, implying that post-training still gains efficiency, but now pays for it with lost coverage.
**Figure 23** Per-task scaling curves on WebShop for the smallest and largest Gemma-4 backbones (gemma-4-E4B, top two rows; gemma-4-31B, bottom two rows). In each block, the top row shows five base (teal) and five post-trained (orange) task curves from the harder (5–25%), intermediate (40–60%), and easier (75–95%) rank quantiles. Tasks are ranked separately for each policy by their success counts, among the tasks that at least one policy solves at least once. The bottom row shows the density of all 500 per-task curves of each policy and their mean pass@k, with pass@1 and pass@128 given in the legend. All curves use the unbiased estimator from 128 rollouts, and a flat curve at zero means no observed success.
**Figure 24** Per-task scaling curves on BFCLv4 for the smallest and largest Gemma-4 backbones (gemma-4-E4B, top two rows; gemma-4-31B, bottom two rows). In each block, the top row shows five base (teal) and five post-trained (orange) task curves from the harder (5–25%), intermediate (40–60%), and easier (75–95%) rank quantiles. Tasks are ranked separately for each policy by their success counts, among the tasks that at least one policy solves at least once. The bottom row shows the density of all 200 per-task curves of each policy and their mean pass@k, with pass@1 and pass@128 given in the legend. All curves use the unbiased estimator from 128 rollouts, and a flat curve at zero means no observed success.
**Figure 25** Global temperature scaling trades single-sample accuracy for coverage rather than improving both. pass@k of the post-trained model of the largest backbone in each family (columns) on BFCLv4 MT, WebShop, and ACEBench (rows), at the default temperature of each benchmark (0.4 for BFCL and 0.7 for the others) and at T ∈ {1.0, 1.3}. Shaded bands show ±1 bootstrap standard error over tasks (200 resamples). A higher temperature usually improves pass@k at larger budgets, but pass@1 drops in some cases.
**Figure 26** Sharpening Tax as a function of the rollout budget for all 14 base/RL pairs. $Tax_A(k)$ (top) and $Tax_S(k)$ (bottom) on BFCLv4 MT, WebShop, and ACEBench, with one curve per checkpoint pair and shaded 95% bootstrap confidence intervals. Positive values mean that post-training reduces test-time scalability relative to the base model. The raw tax starts near zero and grows with the budget for almost every pair, while the calibrated tax can either rise or fall with the budget. Both variants stay negative at $k = 128$ only for Ministral-3-3B, Qwen2.5-3B, and Qwen2.5-7B on BFCL, where post-training repairs the tool-call formatting of the base model.
These results collectively show that while post‑training can boost early success for small models, it induces a coverage‑accuracy trade‑off for larger models that cannot be fixed by a single temperature adjustment.
Sharpening Tax Results
Tax‑guided routing recovers most of the coverage lost to post‑training.
Tax‑based routing improves pass@128 over the post‑trained model by up to 13.6 points.
Figure 33 shows the pass@128 of tax‑based routing exceeding the post‑trained baseline in five of six benchmark‑backbone panels.
Table 7 confirms that post‑trained models achieve higher coverage $C$ and lower raw Sharpening Tax than their base counterparts in every one of the 42 evaluations.
Table 8 breaks $Tax_A(128)$ into three components: “lost” (tasks solved only by the base model), “gained” (tasks solved only by the post‑trained model), and “shape” (differences on tasks solved by both).
The calibrated tax $Tax_S(K)$ aggregates per‑task raw scalability differences, highlighting that high‑tax tasks become deterministic after post‑training, while low‑tax tasks retain the rare successes of the base model.
**Figure 33** Tax-guided routing between the base and post-trained policies. Success rates averaged over the four largest (top) and the four smallest (bottom) backbones on BFCLv4 MT, WebShop, and ACEBench. Each task is served by one policy chosen by one of five rules, i.e., base only (teal), post-trained only (orange), uniform random routing (gray), tax-based routing (red), and optimal routing (navy). Tax-based routing chooses the policy for each task based on its $Tax_S(32)$ estimated from 32 early pilot rollouts. Optimal routing is an oracle selector that assigns each task to the better policy on the full 128 rollouts, which upper-bounds any router. Filled bars show pass@128 and dashed inset bars show pass@1 under the same assignments, both computed from all 128 rollouts of the selected policy including the pilot rollouts. Error bars show the bootstrap uncertainty of pass@128 (2,000 resampled tasks).