TPPO: Tracking-Value PPO for Language Model Reasoning

Futing Wang, Chenxi Li, Jianhao Yan, Yun Luo, Zhi Wang, Yizhuo Li, Runzhe Zhan, Yu Cheng, Tao Lin

TPPO improves language model reasoning by parameterizing the critic as a cumulative sum of log-odds increments.

How can we improve credit assignment in language model reasoning by forcing the critic to explicitly track value evolution across tokens?

Standard reinforcement learning critics struggle to assign credit in long reasoning chains because they estimate prefix values independently, failing to capture how success probability shifts token-by-token. Tracking-Value PPO (TPPO) models this evolution by parameterizing the critic as a cumulative sum of position-wise scalar outputs in log-odds space, which are trained against terminal outcomes. This approach consistently outperforms standard PPO on mathematical reasoning and coding tasks, improving average accuracy by up to 10.1 percentage points.

Paper Primer

In long-chain reasoning, a model might make a correct step followed by an incorrect one, yet standard critics often assign similar values to both prefixes. Because these critics predict each state value independently, they fail to track the local "value evolution"—the change in success probability—that determines whether a specific token deserves more or less credit than its predecessor.

TPPO treats value evolution as a Bayesian update: the transition between successive prefix values is additive in log-odds space. The method implements this by parameterizing the critic as a cumulative sum of learned scalar increments, effectively forcing the model to explicitly learn the local changes in success probability rather than just the absolute values.

TPPO significantly improves reasoning performance across diverse benchmarks.

On Qwen3-8B-Base mathematical reasoning, TPPO achieved an average accuracy improvement of 8.4 percentage points over PPO.

The cumulative parameterization leads to more reliable credit assignment.

Diagnostic analysis shows that TPPO's advantage estimates achieve 15.3–19.9 percentage points higher sign agreement with Monte Carlo-based reference advantages compared to standard PPO.

Why does this approach work better than simply training a standard critic with a better loss function?

Ablation studies show that while using Binary Cross-Entropy (BCE) instead of Mean Squared Error (MSE) provides some benefit, the primary performance gain comes from the cumulative parameterization itself, which explicitly models the additive structure of value evolution.

Does this method require extra computation or process-level labels?

No. TPPO uses the same terminal-outcome supervision as standard PPO and requires no additional rollouts, search, or intermediate process-level annotations.

By explicitly parameterizing the local changes in value rather than absolute state values, TPPO provides a more stable and accurate credit signal for long-horizon reasoning tasks without increasing training overhead.

Introduction

We expose why PPO critics miss token‑level value changes and set up the need for TPPO.

Standard PPO critics estimate a value for each reasoning prefix in isolation, so they never see how the value should shift as each new token is added. This omission makes token‑level credit assignment unreliable, especially under sparse terminal rewards where the true value change can be large and even opposite in sign. The resulting mismatch leads to systematic sign errors and degrades advantage estimates, as shown by the drop in sign‑accuracy from 50.34 % to 45.91 % during training.

Standard PPO critics treat each prefix as an independent prediction, so they cannot capture the incremental change in value that occurs when a new token is appended.

The gap between token‑level credit and critic estimation is the core obstacle this work addresses.

The Critic Credit Assignment Problem

Standard PPO critics miss how value changes across successive tokens, leading to poor credit assignment.

PPO critics predict an absolute value for each reasoning prefix independently, but they do not model how that value evolves as tokens are added. Consequently, the advantage signal used for policy updates can mis‑order token credit, degrading learning stability.

The generation process can be viewed as a Markov decision process where each token extends the current prefix, forming a new state.

How does this MDP differ from a standard language model?

In a language model the next‑token distribution is learned without an explicit notion of state or reward; here each token choice is an action in a state that includes the full prefix, and credit must be propagated back from the final binary reward.

PPO trains a policy together with a separate value network that predicts the expected return from the current prefix.

Why does PPO use a separate value network instead of reusing the policy network?

The value network provides a low‑variance estimate of the expected return, which stabilizes the advantage calculation; mixing it with the policy network would entangle policy updates with value estimation and increase variance.

GAE blends multi‑step temporal‑difference errors to produce a smoother advantage estimate, trading bias for lower variance.

What happens if $\lambda=0$ or $\lambda=1$ in GAE?

With $\lambda=0$ GAE collapses to the one‑step TD error, giving high bias but low variance; with $\lambda=1$ it becomes the full Monte‑Carlo return, yielding low bias but high variance.

**Figure 1.** Critics mispredict how value evolves along reasoning. (a) Representative responses showing that different steps within a response can cause different MC-estimated value changes; the PPO critic predicts changes with opposite signs in these examples. (b) The sign accuracy of these predictions decreases during training. MC evaluation details are in Appendix D.3.

Standard PPO critics fail to capture incremental value changes, so token‑level credit assignment becomes noisy and unstable.

The TPPO Architecture

TPPO reparameterizes the critic to accumulate token‑wise logits, enabling stable tracking of value evolution.

Standard PPO critics predict a value for each prefix in isolation, which ignores how values change as tokens are added. This omission makes credit assignment noisy and hampers learning stability.

When a token is appended, the true value update is an additive change in log‑odds; TPPO builds the critic to mirror exactly that additive pattern.

Compute the first increment: $c^1 = \operatorname{logit}V(s_1) = -0.41$.

Second increment: $c^2 = \operatorname{logit}V(s_2) - \operatorname{logit}V(s_1) = -0.20 - (-0.41) = 0.21$.

Third increment: $c^3 = \operatorname{logit}V(s_3) - \operatorname{logit}V(s_2) = 0.85 - (-0.20) = 1.05$.

Accumulate: $z_1 = c^1 = -0.41$, $z_2 = c^1 + c^2 = -0.20$, $z_3 = c^1 + c^2 + c^3 = 0.85$.

Apply sigmoid: $V(s_i)=\sigma(z_i)$ reproduces the original probabilities (up to numerical rounding).

The additive log‑odds view makes each token’s contribution explicit; learning $c^i$ lets the model reconstruct the full value trajectory from tiny scalar pieces.

How does TPPO’s update differ from a standard critic that predicts $V(s_t)$ directly?

Standard critics output a single probability per prefix, so the model must implicitly learn the whole trajectory. TPPO forces the model to output a separate scalar for each token and then sums them, matching the true additive log‑odds structure.

Instead of a single head that spits out a value for every prefix, TPPO uses a shared backbone that emits a tiny scalar at each position; the prefix value is the sigmoid of the cumulative sum of those scalars.

**Figure 2.** Difference between standard and TPPO critic parameterizations. (a) A standard critic directly predicts a value for each prefix. (b) TPPO accumulates token-wise critic outputs into value logits, then applies a sigmoid to obtain prefix values.

After token 1: $z_1=0.2$, $V_1=\sigma(0.2)\approx0.55$.

After token 2: $z_2=0.1$, $V_2=\sigma(0.1)\approx0.52$.

After token 3: $z_3=0.4$, $V_3=\sigma(0.4)\approx0.60$.

After token 4: $z_4=0.4$, $V_4$ remains $\approx0.60$ (no change because $f_\phi(s_4)=0$).

The model can increase, decrease, or hold the value simply by adjusting the sign of each scalar; this flexibility is unavailable when a single head predicts each $V_t$ independently.

Why can’t a standard critic emulate the same additive behavior by simply learning a different non‑linear mapping?

Because a single‑head network must produce the final probability directly; it has no internal state that persists across positions, so it cannot guarantee that the difference between successive logits equals a learned token‑specific term. TPPO’s explicit accumulation enforces that property by construction.

Empirical Evaluation

TPPO’s empirical gains over PPO across reasoning and coding tasks.

Standard PPO critics predict absolute values for each reasoning prefix, missing the incremental evolution between steps; TPPO instead tracks these changes, yielding more stable credit assignment. We evaluate three questions: (RQ1) does TPPO improve policy performance? (RQ2) does it track value evolution better? (RQ3) does it produce higher‑quality advantage estimates?

GRPO is a PPO variant that adds a gradient‑regularization term to the loss, aiming to reduce policy variance.

How does GRPO differ from standard PPO?

Standard PPO only clips policy ratios; GRPO adds an explicit penalty on the gradient magnitude, so updates are both clipped and regularized, which can curb abrupt policy swings.

TPPO outperforms PPO across reasoning and coding tasks.

Table 2 shows average accuracy gains of +7.0 % (4 B) and +8.4 % (8 B); Table 4 reports coding improvements up to +15.5 % on LiveCodeBench; Figure 3 demonstrates a sustained validation‑accuracy lead.

**Table 2.** In-domain and out-of-domain results with $n = 8$. All entries are accuracies (%). Orange parenthetical values show the absolute improvement over PPO in percentage points. Bold and underlined values indicate the best and second-best results within each model group, respectively.

**Figure 3.** Training dynamics comparing PPO and TPPO on Qwen3-8B-Base ($n = 8$). We track actor gradient norm, policy entropy, validation accuracy, and Pass@4 over training. TPPO establishes an early lead over PPO in both performance metrics and retains it through the final checkpoint.

**Table 3.** In-domain and out-of-domain results with $n = 1$. All entries are accuracies (%). Orange parenthetical values show the absolute improvement over PPO in percentage points. Bold and underlined values indicate the best and second-best results within each model group, respectively.

**Figure 5.** **Value evolution tracking.** Sign agreement between critic predictions and MC-estimated value changes for SNR > $\tau$. Results use Qwen3-8B-Base mathematical reasoning with $n = 8$.

**Figure 6.** **Credit assignment.** Cosine similarity and sign agreement with GAE advantages computed from MC values. Results use Qwen3-8B-Base mathematical reasoning with $n = 8$.

**Figure 4.** Countdown task on Qwen2.5-Math-1.5B. Trained on problems with three or four input numbers.

**Table 4.** Coding results on Qwen3.5-9B. Accuracy (%) averaged over three evaluation runs. Orange parenthetical values show the improvement over PPO in percentage points. HumanEvalFix averages Python, Rust, and JavaScript.

**Figure 7.** Value estimates on test data. Error and accuracy against MC estimates at different reasoning positions.

TPPO consistently outperforms PPO across math and coding tasks, suggesting that tracking value evolution is a broadly useful improvement for RL‑based language models.

Limitations

We outline TPPO’s scope limits and analyze value‑function learning and credit assignment.

Our empirical evaluation uses models up to 9 B parameters, so the scalability of TPPO beyond that size remains unverified. We also test only three domains—mathematical reasoning, Countdown, and coding—leaving broader task generality open. The critic is evaluated by sampling Monte‑Carlo continuations at a few anchor points, which trades trajectory coverage for per‑anchor accuracy and can miss value fluctuations between anchors; finite samples add estimation noise.

The current formulation assumes binary terminal rewards and a discount factor $\gamma=1$. Extending TPPO to dense or non‑binary rewards would require additional investigation.

We formalize the value function learning objective as $L(V)=\mathbb{E}_{\tau\sim\pi}\!\big[(V(s_t)-G_t)^2/2\big]$, whose population minimizer satisfies $V^{*}(s_t)=\mathbb{E}_{\pi}[G_t\mid s_t]=V_{\pi}(s_t)$. For terminal‑only rewards this reduces to $V_{\pi}(s_t)=\Pr_{\pi}(R(y)=1\mid s_t)$, i.e., the critic estimates the success probability of continuations.

In autoregressive generation the token‑level advantage equals the change in prefix value: $A_{\pi}(s_t,a_t)=V_{\pi}(s_{t+1})-V_{\pi}(s_t)=\Delta V_{\pi,t}$. The GAE coefficient $\lambda_{\text{actor}}$ interpolates between the one‑step change ($\lambda=0$) and the full‑trajectory return ($\lambda=1$). For $0<\lambda_{\text{actor}}<1$, future value changes are geometrically weighted, emphasizing nearby changes and attenuating distant terminal signals.

When $\gamma=\lambda=1$, the advantage estimator simplifies to $A_t=R(y)-V_t$, and its expectation recovers the true token‑level value change. Consequently $A_t-A_{t+1}= \Delta V_t$, linking adjacent‑token credit directly to the local value change.

To visualize value evolution we evaluate states at fixed fractions $\rho_j$ of the response length, mapping to token indices $t_j=\lfloor\rho_j T\rfloor$. The coarse‑grained change $\Delta^{(\rho)}_j V = V(s_{t_{j+1}})-V(s_{t_j})$ aggregates token‑level changes over the segment.

Monte‑Carlo estimates $V^{\text{MC}}(s_{t_j})=\frac{1}{K_j}\sum_{k=1}^{K_j}R(y^{(k)})$ provide a sampled reference for $V_{\pi}(s_{t_j})$. Their changes $\Delta^{(\rho)}_j V^{\text{MC}}$ can be compared to PPO‑predicted changes $\Delta^{(\rho)}_j V^{\text{PPO}}$ to assess ordering consistency via the AGAE identity $\text{AGAE}_{t_j}[V]-\text{AGAE}_{t_{j+1}}[V]=\Delta^{(\rho)}_j V$.

**Figure 8.** **Critic ablations and actor GAE coefficients.** Validation accuracy for (a) critic parameterization and loss variants, and (b) PPO and TPPO with $\lambda_{actor} \in \{1, 0.95, 0\}$. Settings for (b) are given in Appendix E.2.

Discussion

We discuss empirical findings, ablations, and broader implications, then acknowledge support.

Table 4 reports coding accuracy on Qwen3.5‑9B. TPPO consistently exceeds PPO, with the largest gains on the HumanEvalFix benchmark (+8.5 pp) and a 15.5 pp improvement on SciCode.

Ablations isolate the source of these gains. Adding a binary cross‑entropy (BCE) head to a standard PPO critic (PPO + BCE) improves over PPO, but TPPO still achieves higher validation accuracy, confirming the importance of the cumulative parameterization. Replacing BCE with MSE in TPPO (TPPO + MSE) degrades performance, showing BCE’s additional benefit.

Modeling value evolution improves the critic’s value estimates. Compared to Monte Carlo (MC) returns, TPPO exhibits lower mean absolute error and higher value‑accuracy at every evaluated position, even though error grows toward the response end.

Varying the actor’s GAE coefficient $\lambda$ tests whether TPPO’s advantage persists across credit‑assignment regimes. Across $\lambda\in\{0,0.5,1\}$, TPPO retains its performance edge over PPO, indicating that better modeling of value evolution benefits credit assignment regardless of how far advantages propagate.

In summary, TPPO introduces a cumulative parameterization that tracks value evolution, leading to more stable critics, improved value estimates, and superior credit assignment without extra rollout sampling or process‑level labels.

We thank the Shanghai Artificial Intelligence Laboratory for support and acknowledge the open‑source communities whose contributions made this work possible.

Theoretical Proofs

Formal derivations of the Bayesian prefix‑value update and the learning signal for cumulative parameterization.

This appendix supplies the formal derivations underlying the Bayesian prefix‑value update (Section C.1) and the suffix‑aggregated learning signal for the cumulative parameterization (Section C.2).

Experimental Details

Provides training configurations, additional results, and ablation analyses.

This appendix details the experimental setup, reports extra results, and presents ablation studies that probe design choices such as output‑head parameterization and actor‑GAE coefficients.

**Table 5.** Training, rollout, and optimization settings. Shared settings apply to both domains unless a domain-specific value is shown.

**Figure 9.** **Additional results on Llama3.2-3B-Instruct.** Overall scores for the base model, PPO, GRPO, and TPPO (%).

**Figure 10.** Value-change sign accuracy by rollout source. Rows distinguish PPO-actor and TPPO-actor rollouts; columns show the SNR thresholds $\tau = 0$ and $\tau = 1$.

Diagnostic metrics evaluate how well critics predict incremental value changes. For each adjacent MC anchor we compute $\Delta$`MC_j` = VMC_{j+1} − `VMC_j` and its signal‑to‑noise ratio `SNR_j` = |$\Delta$`MC_j`| / √($s_j$² / $n_j$ + $s_{j+1}$² / $n_{j+1}$). Sign accuracy at threshold $\tau$ is Pr[sign($\Delta$`V_j`) = sign($\Delta$`MC_j`) | `SNR_j` > $\tau$].

**Table.** MC-based reference advantages. For these diagnostics, $\gamma = \lambda = 1$, intermediate rewards are zero, there is no KL reward shaping, and the value after termination is zero. GAE therefore reduces to the terminal outcome minus the value at the anchor:

**Table.** Separate-head parameterization. We also tested a variant with two independent scalar heads,

**Figure 11.** Shared and separate output heads. Validation accuracy, response length, and actor gradient norm for PPO, the shared-head TPPO critic, and the two separate-head variants shown in the legend.

In the actor‑GAE ablation (E.2) we vary $\lambda_{a}$ctor ∈ {1, 0.95, 0} while keeping the critic’s $\lambda$ = 1. Setting $\lambda_{a}$ctor = 0 corresponds to segment‑level advantage estimation without intermediate rewards, mirroring the VinePPO approach.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers