TTPO: Test-Time Policy Optimization

Aozhe Wang, Zhengxi Lu, Jianze Wang, Shangke Lv, Ying Liu, Weiming Lu, Jun Xiao, Yueting Zhuang, Hua Yang, Qianglong Chen, Yongliang Shen

TTPO uses an asymmetric objective to enable label-free test-time training by distilling agreeing rollouts and penalizing disagreeing ones.

How can we improve language model reasoning at test-time without access to ground-truth labels?

Large language models struggle to improve during test-time training because they lack ground-truth labels, and relying on majority-vote pseudo-labels often propagates errors when the consensus is wrong. The authors propose Test-Time Policy Optimization (TTPO), which treats rollouts asymmetrically: it distills trajectories that agree with the pseudo-label toward an answer-conditioned teacher, while applying reinforcement learning penalties to those that disagree. This approach matches or exceeds label-supervised performance on competition-level benchmarks, raising a 1.7B model's accuracy from 38.0% to 45.2% without any ground-truth supervision.

Paper Primer

TTPO hinges on the observation that pseudo-label errors are asymmetric: while distillation toward a wrong pseudo-label corrupts the model, penalizing a rollout simply for disagreeing with the majority remains correct in the vast majority of cases. The method acts like a filter: it uses the pseudo-label as a guide for positive reinforcement, but relies only on the fact of disagreement for negative reinforcement.

TTPO achieves label-free performance parity with label-supervised self-distillation.

Across five competition-level benchmarks, TTPO (using only pseudo-labels) matches or exceeds the performance of OPSD (using ground-truth labels) on Qwen3-1.7B, 4B, and 8B models. On Qwen3-1.7B, TTPO improves average accuracy from 38.0% to 45.2% in a purely label-free test-time training setting.

The asymmetric objective enables robust self-evolution.

Unlike ground-truth routing, which starves the model of gradients when the model's performance is low, majority-vote routing ensures a consistent supply of positive and negative samples, allowing the model to improve its own pseudo-label quality over time. TTPO gains +25.2% to +36.4% in accuracy when evaluated with thinking mode disabled, significantly outperforming standard self-distillation.

Why is this approach more robust than simply using majority-vote pseudo-labels as ground truth?

Distilling toward a wrong pseudo-label forces the model to learn incorrect reasoning at every token. By contrast, penalizing disagreeing rollouts via reinforcement learning only requires the model to recognize that a trajectory is "not the majority," which remains a valid signal even when the majority itself is incorrect.

What is the primary limitation of this method?

The method's effectiveness is fundamentally bounded by the quality of the initial majority-vote consensus; if the model's base performance is too low to form a meaningful majority, the training signal becomes unreliable.

Motivation and Problem Framing

We expose why label‑dependent test‑time training fails and motivate a label‑free pseudo‑label approach.

Test‑time training (TTT) aims to improve a model on the very problems it must solve, but existing methods—reinforcement learning from coarse sequence‑level rewards and on‑policy self‑distillation (OPSD)—all require ground‑truth answers to supervise updates.

Label‑dependent supervision creates a bottleneck for TTT because no ground‑truth is available at inference time.

OPSD lets a policy teach itself by conditioning a teacher on the correct answer and then distilling token‑wise predictions.

GRPO penalizes rollouts that deviate from a reference policy by comparing groups of trajectories.

When we replace ground‑truth with majority‑vote pseudo‑labels, the pseudo‑label is wrong on roughly 85 % of competition‑level prompts, yet about 79 % of the rollouts that disagree with that pseudo‑label are also wrong, making disagreement a reliable signal even when the pseudo‑label itself is noisy.

Test‑Time Policy Optimization (TTPO) swaps ground‑truth supervision for majority‑vote pseudo‑labels and applies an asymmetric objective that treats agreeing and disagreeing rollouts differently.

**Figure 1.** (a) **Motivation:** When pseudo-labels are wrong, most disagreeing rollouts are genuinely wrong too, during Qwen3-1.7B TTT on AIME 2026. (b) **Overview of TTPO.** (c) **Performance:** Average accuracy of Qwen3-1.7B across AIME 2026, HMMT 2026, and BRUMO 2025.

The shift from ground‑truth reliance to pseudo‑label self‑improvement enables label‑free test‑time training without sacrificing performance.

The TTPO Mechanism

TTPO splits sampled answers into positives and negatives, applying distinct losses to limit label errors.

Test‑time training must adapt without ground‑truth labels, but majority‑vote pseudo‑labels are often wrong, so a naïve self‑distillation would corrupt all trajectories. TTPO mitigates this by treating positive and negative samples asymmetrically.

The method samples $K$ answer trajectories, picks the most frequent answer $\hat{a}$ as a pseudo‑label, and separates the trajectories into positives (agreeing with $\hat{a}$) and negatives (disagreeing). Positives receive a forward‑KL loss, negatives receive a group‑advantage loss, so only the small positive set can be corrupted by a wrong pseudo‑label.

Positive set $P=\{1,2\}$ because $a^{(1)}$ and $a^{(2)}$ equal $\hat{a}$.

Negative set $N=\{3,4\}$ because $a^{(3)}$ and $a^{(4)}$ differ from $\hat{a}$.

Compute $\text{LOPSD}(1)$ and $\text{LOPSD}(2)$ using forward KL on their tokens.

Compute $\text{LGRPO}(3)$ and $\text{LGRPO}(4)$ using group advantage and token masking.

Aggregate the four losses with $\lambda$ and take a gradient step.

Even if $\hat{a}=5$ is wrong, only the two positive trajectories are exposed to the wrong teacher; the two negatives are updated by a label‑free signal.

How does TTPO differ from ordinary self‑distillation that applies the same loss to all sampled trajectories?

Ordinary self‑distillation treats every trajectory as if the pseudo‑label were correct, so a wrong label corrupts all $K$ samples. TTPO isolates the potentially corrupted set (the positives) and uses a label‑free group‑advantage loss for the negatives, dramatically reducing error spread.

The pseudo‑label $\hat{a}$ is obtained by clustering the extracted answers and selecting the largest cluster; this cluster’s size $c$ measures consensus.

Cluster answers: $\{12,12\}$, $\{13\}$, $\{15\}$.

Select the cluster $\{12,12\}$ as the majority.

Set pseudo‑label $\hat{a}=12$ and record $c=2$.

Even though the true answer might be $13$, the method proceeds with $\hat{a}=12$, and the subsequent asymmetric loss limits damage.

Why can the majority‑vote pseudo‑label be wrong so often?

Because the test‑time problems are extremely challenging; the model often produces diverse incorrect answers, and the most frequent wrong answer can dominate the cluster even when the ground‑truth answer is unique.

Instead of applying the same loss to every trajectory, TTPO uses forward KL only on positives (which may be corrupted) and a group‑advantage loss on negatives, which does not depend on the pseudo‑label content.

Why not simply apply forward KL to all $K$ trajectories as in standard self‑distillation?

Applying forward KL to all trajectories would propagate the teacher’s error to every sample when $\hat{a}$ is wrong. The asymmetric objective limits this propagation to the positive set, which is typically much smaller.

TTPO weights or masks individual tokens based on how much learning signal they carry: tokens where the student is already confident and aligned receive low weight, while uncertain or misaligned tokens receive high weight.

Compute $w(1)=0.2+0.3-0.2\cdot0.3=0.44$.

Compute $w(2)=0.8+0.4-0.8\cdot0.4=0.88$.

Compute $w(3)=0.1+0.2-0.1\cdot0.2=0.28$.

Compute $w(4)=0.5+0.6-0.5\cdot0.6=0.8$.

These weights modulate the KL loss, emphasizing tokens 2 and 4.

Tokens with either high uncertainty or high divergence dominate the loss, while a token that is both certain and aligned (token 3) receives little influence.

Why does TTPO use a soft‑OR combination for token weights instead of a simple average?

The soft‑OR ensures that a token receives a high weight if *either* the student is uncertain *or* the teacher disagrees strongly, preserving learning signal when either condition signals a need for correction.

**Figure 2.** Overview of TTPO. (1) Majority Voting: $K$ sampled trajectories are partitioned into positive ($a_k = \hat{a}$) and negative ($a_k \neq \hat{a}$) sets. (2.1) OPSD Branch: positive samples are supervised via per-token forward KL with token weighting. (2.2) GRPO Branch: negative samples are penalized via group advantages with token masking on anomalous positions.

With these components—majority voting, asymmetric losses, and token‑level selection—TTPO can improve test‑time performance without any ground‑truth labels.

Empirical Performance

TTPO matches or exceeds label‑dependent baselines across scales without using ground‑truth labels.

TTPO outperforms label‑dependent baselines across all three model scales while using only majority‑vote pseudo‑labels.

Table 1 shows TTPO’s average scores of 40.1, 58.6, 62.6 versus OPSD’s 39.7, 58.4, 61.7.

The privileged‑information study (Table 4) shows that a short answer from a thinking‑mode teacher yields the best trade‑off: it provides enough guidance without overwhelming the student, whereas full trajectories dilute the teacher’s reasoning signal.

**Figure 3.** Ablations over update strategies on Qwen3-1.7B AIME26 TTT setting. $\mathcal{P}$ and $\mathcal{N}$ denote the loss applied to positive and negative samples, respectively. Dotted lines indicate the corresponding variants with an unconditioned teacher.

**Figure 4.** Cross-benchmark generalization (Qwen3-1.7B). Each subplot corresponds to a training benchmark; each group within a subplot shows performance on a different target benchmark.

TTPO achieves competitive performance without ground‑truth labels.

Ablations and Training Dynamics

Ablation studies reveal how each component shapes training dynamics and final performance.

Training dynamics expose a stark contrast: TTPO with ground‑truth (GT) labels barely moves its loss, often stalling near zero, whereas TTPO that relies on majority‑vote pseudo‑labels drives a steady, substantial loss decline. OPSD (Leakage) and OPSD‑TTT also show smooth loss reductions, confirming that vote‑based routing keeps both branches active.

When the thinking mode is disabled at test time, TTPO transfers reasoning ability far more effectively than OPSD. Across model scales, TTPO adds +25.2, +30.6, and +36.4 points respectively, while OPSD only gains +7.1, +5.8, and +3.5 points.

TTPO improves non‑thinking performance by up to +36.4 points over the base model.

Table 7 shows TTPO gains of +25.2 (1.7 B), +30.6 (4 B), and +36.4 (8 B) points, far exceeding OPSD’s modest improvements.

**Figure 5.** **Left:** Comparison between pseudo-label vs. ground-truth supervision. TTPO w/ GT replaces majority-vote pseudo-labels with ground truth while keeping the asymmetric objective and token-level selection intact; OPSD (Leakage) trains standard OPSD directly on AIME26. **Right:** Entropy during training. Values in parentheses on the x-axis denote TTRL's training steps.

**Figure 6.** Avg@12 and Maj@12 during TTPO TTT on HMMT26 (Qwen3-1.7B). Dashed line: base Maj@12.

**Figure 8.** **Left:** Training loss curves for the OPSD and GRPO branches (unweighted). **Right:** Positive sample fraction over training steps. (Qwen3-1.7B, OpenThoughts).

Setting the RL weight to $\\lambda\\!=\\!0.10$ yields the best overall scores (46.5 AIME26, 31.6 HMMT26, 54.7 BRUMO25).

Table 8 reports peak performance at $\\lambda\\!=\\!0.10$, where the GRPO and OPSD losses are balanced.

A fixed 50/50 positive‑negative split achieves the highest benchmark results (46.5, 31.6, 54.7).

Table 9 shows the “Fixed (0.5)” row outperforming both random and dynamic fractions.

Selecting the shortest completions maximizes accuracy (46.5 AIME26, 31.6 HMMT26, 54.7 BRUMO25).

Table 10 highlights the “Shortest” strategy as the top performer across all three benchmarks.

Theoretical Foundations

Analyzes the KL‑based training signal that lets TTPO improve without ground‑truth labels.

Test‑time training must update the model without any ground‑truth supervision, which makes the learning signal fragile. The KL‑based term provides a principled way to push the policy toward the pseudo‑label while exposing when that push becomes harmful.

The KL term measures how the policy that sees the pseudo‑label diverges from the original policy; minimizing it nudges the model to behave as if the pseudo‑label were true.

Compute the weighted log‑probabilities of $p_t$: $\log p_t(v_1)=\log0.4$, $\log p_t(v_2)=\log0.4$, $\log p_t(v_3)=\log0.2$.

Form the gradient contribution $-q_t(v)\,\nabla_{\theta}\log p_t(v)$ for each $v$: $-0.7\,\nabla_{\theta}\log0.4$, $-0.2\,\nabla_{\theta}\log0.4$, $-0.1\,\nabla_{\theta}\log0.2$.

Sum the three contributions to obtain $\nabla_{\theta}\mathrm{KL} = -0.7\,\nabla_{\theta}\log0.4 -0.2\,\nabla_{\theta}\log0.4 -0.1\,\nabla_{\theta}\log0.2$.

Because $v_1$ (the pseudo‑label) has the largest weight, the update increases $p_t(v_1)$ and decreases $p_t(v_2),p_t(v_3)$, moving the baseline toward the pseudo‑label.

When the pseudo‑label matches the true answer, the KL gradient reinforces the correct action; when it does not, the same formula injects a conflict term that can pull probability away from the true answer.

How does this KL‑based signal differ from a standard policy‑gradient update that uses ground‑truth rewards?

Standard policy gradient computes an advantage for the single true action and updates $p_t$ directly toward that action. The KL signal instead takes an expectation over the full distribution $q_t$ induced by the pseudo‑label, so every action is weighted by how likely the pseudo‑label‑conditioned policy thinks it is. This yields a softer, distribution‑level push and introduces the conflict term $\Delta_{\text{conflict}}(t)$ when the pseudo‑label is wrong.

Because the KL signal relies on majority‑vote quality, it degrades when the rollout budget $K$ is tiny or the problem is so hard that no rollout yields the correct answer. In such regimes both the positive and negative branches receive noisy supervision, limiting the method’s applicability beyond well‑structured mathematical tasks.

Prompt Templates and Caveats

We visualize token weighting and masking on positive and negative samples to show their complementary effects.

The section begins by showing the two prompt templates used for the student and the teacher models. The student prompt disables thinking mode and simply asks the model to solve the problem step‑by‑step, enclosing the final answer in a box. The teacher prompt enables thinking mode, provides a reference solution, and instructs the model to re‑derive the answer independently.

In the positive sample (answer 7 ✓) token weighting down‑weights low‑entropy, low‑divergence tokens such as the deterministic coordinates $(0, a)$, $(0, 0)$, and $x = 0$. High weight is assigned to tokens that convey geometric insight, e.g., the statement that the extension beyond $A$ lies on the line $x = 0$ and the conclusion that $K$ is at $(0, y)$ with $y < 0$. These tokens are where the student model is uncertain, providing the primary learning signal.

The underlying geometry problem asks for $LD$ given a square $ABCD$ with $L$ on $CD$, $K$ on the extension of $DA$ beyond $A$, a right angle at $KBL$, $KD = 19$, and $CL = 6$, with the ground‑truth answer $LD = 7$.

In the negative sample (answer 397 ✗) token masking suppresses tokens that are locally correct, such as the arithmetic $s$ and the equality $= 6$, while keeping tokens that reveal the reasoning error. The kept tokens include the mistaken claim that $CD$ is vertical and the resulting erroneous coordinate $L = (s, s - 6)$, which drive the penalty toward the flawed geometric inference. Masking thus isolates the confident mistakes without harming the model’s correct sub‑skills.

Together, token weighting and token masking implement a dual philosophy: weighting highlights where the model still needs to learn, whereas masking highlights where the model is confidently wrong. By avoiding gradient updates on low‑signal tokens from opposite directions, the two mechanisms yield more efficient and stable test‑time training.

Implementation and Configuration

Implementation details: hardware, optimizer, and training/evaluation settings for TTPO and baselines.

All experiments use the AdamW optimizer (bfloat16) together with Flash Attention 2; TTPO runs on 4 × H100 GPUs while all other methods use 8 × H100 GPUs. We apply full‑vocabulary logit distillation to every distillation‑based method (OPSD, OPSD‑TTT, TTPO) and follow a thinking‑mode‑off student / thinking‑mode‑on teacher regime with LoRA adapters disabled. For TTPO and OPSD‑TTT the maximum sampling length is capped at 16 000 tokens, but gradient updates affect only the first 1 024 completion tokens.

**Table 5.** Training configuration for all methods. GRPO and OPSD use ground-truth labels; TTRL, OPSD-TTT, and TTPO are label-free. “–” indicates not applicable.

**Table 6.** Evaluation configuration.

**Figure 7.** Training loss curves.

**Table 7.** Non-thinking evaluation on OpenThoughts training data. Models are evaluated with thinking mode disabled. Both OPSD and TTPO are trained with a thinking-mode-on teacher.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers