Self-Retrospection Distillation: Turning Post-Hoc Experiences into Prior Foresight

Haoxiang Zhang, Qinglin Chen, Hiroaki Hayashi, Zhuofeng Li, Siming Zhang, Jiaxin Zhang, Jixuan Chen, Fang Wu, Pan Lu, Silvio Savarese, Julian McAuley, Chien-Sheng Wu

Self-Retrospection Distillation (SRD) uses post-hoc agent experience to supervise pre-interaction foresight, bypassing reward-uniform blind spots.

How can we improve reinforcement learning in reward-sparse environments by distilling post-hoc trajectory analysis into prospective foresight?

Reinforcement learning with verifiable rewards (RLVR) relies on outcome contrast to improve agent policies, but this signal vanishes when all rollouts in a group receive the same reward. Self-Retrospection Distillation (SRD) solves this by distilling post-hoc trajectory information into a trajectory-blind foresight model, training the agent to anticipate task demands and failure modes before it acts. This approach provides dense supervision even when reward contrast is absent, yielding gains of up to 24.2 percentage points in reasoning and agentic tasks.

Paper Primer

SRD treats completed trajectories as privileged hindsight that supervises a foresight distribution conditioned only on the task and environment. By aligning the token-level distributions of a hindsight-conditioned teacher and a foresight-conditioned student, the agent internalizes patterns of success and failure as anticipatory knowledge.

SRD provides effective learning signals in reward-uniform regimes where standard RLVR fails.

In a 2B parameter setting where 98% of rollout groups were reward-uniform (all-failure), the baseline RLVR achieved 0.0% success, while adding SRD reached 60.6%.

SRD complements existing post-training objectives across diverse task categories.

SRD improved performance across Math, Code, Search, and Agentic benchmarks when applied to GRPO, OPSD, and hybrid RLSD baselines. Gains of up to 24.2 percentage points across evaluated settings.

Why does this approach work when standard reinforcement learning methods fail?

Standard methods require reward variance within a rollout group to generate a gradient; when all rollouts succeed or fail identically, the advantage signal is zero. SRD bypasses this by scoring each trajectory against its own hindsight-conditioned prospection, which remains informative regardless of group-level reward outcomes.

Is this a new architecture or a training-time modification?

It is a training-time auxiliary objective. SRD adds a lightweight loss term to the existing policy optimization, and the foresight predictions are used only as training targets—they do not need to be explicitly generated at inference time.

SRD establishes that post-hoc agent experience is useful for shaping predictive representations before interaction, effectively turning "hindsight" into a reliable source of "foresight" for future tasks.

Introduction and Motivation

We expose why scalar rewards fail in group‑relative RL and propose prospective learning.

Reinforcement learning with verifiable rewards (RLVR) turns agent experience into learning signals primarily through scalar outcome rewards after interaction. For group‑relative objectives, this signal vanishes when all rollouts receive the same reward, even though their trajectories contain rich information about task requirements and failure modes.

We therefore ask whether hindsight can teach an agent what it could have anticipated before acting. Prospective learning uses post‑hoc experience to supervise foresight predictions from the pre‑interaction view, and we instantiate this idea with Self‑Retrospection Distillation (SRD), which aligns token‑level distributions of a pre‑interaction prediction with a hindsight prediction derived from the completed trajectory.

**Figure 1.** Comparison of three paradigms for converting agent experience into training signals. Reinforcement Learning with Verifiable Rewards (RLVR) scalarizes completed trajectories into outcome rewards and assigns them back to the action policy. Retrospective Learning retains structured post-hoc hindsight to supervise behavior. Prospective Learning uses hindsight to supervise predictions formed from information available before interaction.

The failure of scalar rewards in group‑relative objectives leaves most trajectories signal‑free.

Preliminaries

We formalize the RL setting and introduce the GRPO baseline for group‑relative rewards.

We formalize the reinforcement‑learning interaction between policy $\pi$_$\theta$ and environment E, then describe the baseline GRPO method that operates on groups of rollouts.

GRPO normalizes each rollout’s advantage by the group’s reward mean and variance, encouraging agents to improve relative to their peers rather than against an absolute baseline.

Self-Retrospection Distillation

We introduce Self‑Retrospection Distillation, a prospective learning trick that turns hindsight into a teacher.

Group‑relative policy optimization (GRPO) stalls when rollouts provide no reward signal, wasting compute. Prospective learning sidesteps this by training the policy to anticipate outcomes before they occur. The following mechanism implements that idea.

Prospective learning asks the model to guess the future action distribution, then aligns it with the distribution it would have after seeing the actual outcome, whereas retrospective learning only learns from the outcome after it happens.

Compute the KL divergence: $D_{KL}$($p_{\text{hind}$}‖$p_{\text{fore}$})=0.9\log\frac{0.9}{0.4}+0.1\log\frac{0.1}{0.6}\approx0.73.

Average this divergence over many sampled rollouts (here just one) to obtain $L_{\text{pro}$}≈0.73.

Back‑propagate the gradient to increase $p_{\text{fore}$}(a=1) and decrease $p_{\text{fore}$}(a=0).

Even a modest mismatch between foresight and hindsight yields a non‑trivial loss, forcing the policy to move its predictions toward the eventual truth.

How is prospective learning different from standard hindsight training?

Standard hindsight training updates the policy after the outcome is known, using the same distribution for both teacher and student. Prospective learning instead trains the student to match a teacher that already knows the outcome, so the student must anticipate the result before acting.

SRD turns the model’s own post‑hoc analysis into a teacher: before acting, the policy generates a foresight prediction, then a stopped‑gradient copy that sees the actual result (the hindsight teacher) aligns its token‑level outputs with the foresight prediction.

Student foresight for rollout 1 predicts token‑level distribution $p_{\text{fore}$}^{(1)} = [0.6, 0.4] for the two tokens.

Teacher (with hindsight) for rollout 1 produces $p_{\text{hind}$}^{(1)} = [0.9, 0.1] after seeing the correct answer.

Compute Jensen–Shannon divergence for rollout 1: JSD≈0.12.

Repeat for rollout 2 (failed): foresight [0.3, 0.7], teacher [0.2, 0.8], JSD≈0.02.

Sum divergences (0.12 + 0.02) and average over the batch to obtain $L_{\text{SRD}$}≈0.07.

Even though the failed rollout provides a weaker signal, the teacher’s hindsight still guides the student toward better predictions, demonstrating why both knowledge and pitfall views are useful.

Why not use ordinary self‑distillation instead of SRD?

Ordinary self‑distillation uses the current policy as its own teacher, offering no new information. SRD’s teacher is conditioned on the actual outcome (the hindsight context), so it supplies richer, outcome‑aware supervision that the student cannot obtain on its own.

**Figure 2.** Overview of Self-Retrospection Distillation (SRD). The student policy rolls out trajectories and generates anticipated pitfalls and knowledge foresights via a differently prompted call to the same model. Post-hoc experience from the completed trajectory conditions the stopped-gradient self-teacher, whose token-level distributions supervise the student's own foresight rollout.

Experimental Overview

SRD delivers sizable performance gains over baselines across multiple tasks and scales.

Table 1 aggregates the key empirical evidence: SRD raises pass rates across all four benchmark families and both model scales.

SRD adds up to +17.22 pp to the Math average compared with the strongest baseline.

OPSD+SRD lifts the 9B Math average from 39.70 % to 56.92 % (Table 1).

**Table.** SRD broadly improves post-training objectives and transfers beyond training conditions.

Beyond the headline gains, SRD improves code benchmarks (4 B GRPO +8.33 pp, OJBench +8.12 pp) and unseen ALFWorld environments (gains 1.12–13.74 pp), and it rescues the instability of pure self‑distillation, which otherwise falls below the untrained policy on several tasks.

Even when evaluation departs from the training interaction regime—e.g., the longer‑horizon BrowseComp‑Plus settings—SRD still delivers improvements (up to +9.52 pp), demonstrating that prospective supervision remains valuable at higher capability levels.

SRD consistently outperforms baselines across diverse domains.

Foresight Channels

We ablate foresight channels and token reallocations to gauge SRD’s impact.

Recall that SRD supplies dense foresight signals to replace sparse group‑relative rewards, enabling learning even when rollouts show little reward variation.

**Table 2.** **Left:** Prefix and channel choice in self-distillation, OPSD. Prefix is what the OPSD teacher sees; SRD target is the distilled perspective; green/red marks a gain/loss over the raw-trace no-SRD row, and Wall-clock is the rollout time per step. **Top center:** the SRD divergence in each perspective during training, on two different domain-shuffled runs on 4B with RLSD; the band is the local interquartile range of the logged points. **Top right:** which epistemic tokens and the classes they belong to the SRD policy changes. **Bottom center:** the GRPO+SRD displacement from the untrained base, decomposed onto GRPO’s own direction: blue is how much of that direction is kept and red is the orthogonal remainder, per split. The dashed line is unity. **Bottom right:** per-token similarity between each objective’s own update direction and the same objective plus SRD, as a cumulative distribution over held-out positions, each median marked where its curve crosses 0.5. The arrows give how far each policy travelled from the base before and after adding SRD.

When we distill only the pitfalls channel, AIME26 improves by +3.3 pp (4B raw), +1.7 pp (9B raw), +3.3 pp (4B hindsight) and +6.7 pp (9B hindsight), and AMO‑Bench rises by +3.5 pp (4B hindsight). Adding the knowledge channel on top of pitfalls yields no additional gain and even hurts performance: AMO‑Bench drops by 4 pp (4B) and AIME26 falls from 65.42 % to 62.92 % (9B). Moreover, the knowledge channel consumes more rollout time (78–87 % of a training step) without improving the divergence, which stays flat (~0.0045 and ~0.003) compared to the steadily decreasing pitfall divergence (‑47 % and ‑30 %).

Distilling the pitfalls channel delivers consistent gains while the knowledge channel adds cost without benefit.

See the performance deltas described above; the pitfalls channel improves multiple metrics, whereas the knowledge channel shows flat divergence and a net loss.

To understand how SRD reshapes the policy’s language, we measured token‑class frequencies across 5 K multi‑turn rollouts. SRD raises connective prose by +8.6 tokens, explicit planning language by +3.7, and problem‑structure words by +1.4, while reducing inline math notation (‑6.7), punctuation (‑4.3), digits (‑4.1) and variable names (‑3.0). Tool‑call syntax is unchanged (+0.03). These shifts nearly cancel, yielding a net increase of +0.6 tokens per rollout, indicating a reallocation of a fixed token budget rather than a total increase in output.

SRD reallocates token budget toward higher‑level planning language without changing overall token count.

The class‑wise token differences above sum to a net +0.6 tokens per rollout.

The Reward-Silent Regime

Ablations reveal that removing SRD collapses learning signals, while SRD restores performance across model scales.

We isolate the effect of SRD by training four Qwen3.5 scales on the code domain, disabling dynamic sampling, and comparing the same objective with and without SRD.

**Figure 4.** Uniform-reward groups waste most of the rollout at both ends of the capability range: (a) usable share of the budget, (b) overall training group composition.

Removing SRD limits the usable portion of the rollout budget to at most 63 %.

Figure 4(a) reports a maximum usable budget of 63 % for GRPO across all four model scales.

At the 2 B scale, uniform‑reward groups dominate (98 % of groups), causing the baseline to achieve only 1.6 % peak training success and 0 % final success.

Raw counts in the text show 98 % uniform groups and a corresponding 1.6 % peak success for the baseline, with final success at 0 %.

Adding SRD restores training success at the 2 B scale to 60.6 % despite the same uniform‑reward group distribution.

With SRD, the same 2 B budget reaches 60.6 % training success, as reported in the ablation results.

SRD (PITFALL) improves ALFWorld performance from 85.12 to 88.69, a gain of 3.57 points.

Table 3 shows the base RLSD objective achieving 85.12, while adding SRD (PITFALL) raises the score to 88.69.

When SRD is combined with PITFALL + KNOWLEDGE under the GRPO objective, the average score across tasks rises to 75.38, surpassing the GRPO baseline average of 67.75.

Table 3 reports an average of 75.38 for the SRD‑enhanced GRPO variant versus 67.75 for the GRPO baseline.

Policy Distribution Analysis

We ablate SRD’s effect on displacement and test‑time foresight to gauge their impact.

We now ask the two ablation questions: (RQ4) how does SRD reshape the prospection distribution, and (RQ5) does exposing foresight at inference add value.

The displacement distribution describes how a policy’s logits move in parameter space relative to a reference baseline when a new training signal is added.

How does this differ from a simple gradient‑step that also moves along the loss surface?

Unlike a raw gradient, the displacement distribution is measured after the full training update and is expressed in the Fisher metric, which accounts for the curvature of the policy’s probability distribution. The orthogonal part therefore represents a change that cannot be captured by scaling the original gradient alone.

SRD pushes the policy 30 % farther along the GRPO direction than GRPO alone.

Measured on the held‑out code split (626 992 positions), $\alpha$/∥A∥ = 1.30 (Table 2, bottom‑center).

SRD also adds a sizable orthogonal component, amounting to 0.49 of the total displacement magnitude.

Computed as ∥E⊥∥/∥E∥ = 0.49 on the same code split.

For the small RL displacement, SRD more than doubles its size (0.39 → 0.86).

Values reported in Table 2; the ratio is roughly 2.2×.

Turning to inference, we ask whether making the foresight channel explicit at test time improves downstream performance.

Test‑time foresight yields at most a 5.00 pp gain on any checkpoint.

Table 3 shows the largest improvement of +5.00 pp for the GRPO+SRD (PITFALL) arm.

The best checkpoint changes by only 11 rollouts out of 800, i.e., ≈1.4 %.

Four checkpoints differ by 2, 3, 5, and 11 rollouts respectively (Table 3).

**Table 3.** Making foresight explicit at inference, on Qwen3.5-4B-Thinking. Each + test-time foresight row re-evaluates the checkpoint above it with the predicted KNOWLEDGE/PITFALL blocks emitted before acting. Benchmarks, splits and protocol are unchanged from §C.5, and Avg is on the same convention as Table 1. Cell shading is each value’s gain over the base of its own objective, on the scale of Table 1; base rows are unshaded. Bold: best per base objective.

Reward-Silent Regime Analysis

Appendix A details the constraints and behavior of the reward‑silent regime.

Overview: The reward‑silent regime arises when rollouts produce no reward variation, causing GRPO to receive uniform signals and stall learning.

Setting: We consider a batch of $G$ groups, each containing $N$ trajectories, and denote the per‑group reward variance as $\sigma_g^2$.

Budget accounting: The computational budget for estimating group‑relative rewards scales with $O(GN)$, but in the reward‑silent case the effective signal‑to‑noise ratio drops to zero, rendering the budget wasteful.

What group normalization optimizes: It rescales each group’s advantage estimate to have zero mean and unit variance, thereby restoring a usable gradient signal even when raw rewards are constant.

The reward‑silent event: When all trajectories in a group receive identical rewards, the GRPO update reduces to a constant shift, providing no learning direction.

How many groups each direction costs to estimate: Estimating the gradient for a single direction requires observing at least two groups with differing reward statistics; otherwise the direction remains unidentifiable.

A remark on the two channels: The forward‑looking foresight channel supplied by SRD supplies dense gradients, while the backward‑looking reward channel remains silent; together they balance learning progress.

Ablation Study on Lambda

Ablation of the $\lambda$ hyper‑parameter shows its impact on benchmark performance.

The table presents performance metrics across five benchmarks (AIME24, AIME25, LCB-v6, HotpotQA, 2Wiki) and an aggregate "all" column for different values of $\lambda$ (1.0, 0.1, 0.01, 0.001) compared to a "base (step 0)" baseline.

Policy Probing

Brief analysis of how LSRD components affect policy behavior and training dynamics.

The probe is configured to sample actions from the policy and record the resulting foresight signals. Changing the probe placement reveals that LSRD relies on accurate forward‑looking estimates. This setup isolates the contribution of the SRD component.

With LSRD active, the policy outputs an additional foresight channel alongside the usual action logits. Removing LSRD eliminates this channel, causing the policy to revert to standard GRPO behavior. The difference highlights that SRD injects predictive information absent in the baseline.

LSRD updates are applied at the points where the policy would normally receive reward feedback. If the SRD update is omitted, gradients are only derived from sparse group‑relative rewards, reducing learning‑signal density. This shows that SRD provides extra update locations.

SRD introduces a longer RL horizon because the foresight signal propagates across multiple timesteps, while the self‑distillation loss is computed in a single step. Removing SRD collapses the horizon, making the RL step shorter but losing the dense guidance. The trade‑off explains the observed training‑speed differences.

The proportion of SRD samples retained after filtering differs per environment due to variance in reward sparsity. Dropping SRD entirely removes this variability, leading to uniform behavior across domains. This variation indicates that SRD adapts to domain‑specific signal richness.

An earlier reported metric linking SRD retention rate to final performance was later found to be a measurement artifact and retracted. Excluding this statistic does not change the qualitative conclusions about SRD’s impact. The retraction underscores the importance of robust evaluation.

When reward contrast is zero, SRD still generates a non‑trivial foresight signal, sharpening the agent’s predictions. Removing SRD in this regime eliminates the signal, confirming that SRD supplies information beyond raw rewards. This demonstrates SRD’s role as a proxy for missing reward gradients.

SRD enforces an implicit contract: the policy must produce actions that are consistent with its own foresight predictions. If SRD is removed, this contract disappears, and the policy may diverge from its anticipated outcomes. The contract explains why SRD improves stability.

In the agent environment, SRD grounds foresight by conditioning on the agent’s own state transitions. Omitting SRD breaks this grounding, causing foresight to become misaligned with actual dynamics. The effect highlights the necessity of interaction‑aware distillation.

SRD leverages groups where all rollouts succeed to populate the KNOWLEDGE channel, enriching the learning signal. Without SRD, this channel remains empty, reducing the amount of useful information. This shows that SRD extracts value from otherwise ignored successes.

During training, the foresight strength of the SRD channels evolves, initially growing rapidly then plateauing. Removing SRD freezes this evolution, leaving the policy with static, weaker signals. The trajectory illustrates SRD’s contribution to progressive learning.

Gradient Estimation Costs

Limits of reward‑based updates: mixed outcomes gate cost, prospective updates avoid the gate.

Section A.1 frames the central question of this limits analysis: which trajectory groups can generate a gradient and what cost is incurred to estimate its direction. The answer underpins RQ 3, probing the empirical impact of the structural gate imposed by mixed‑outcome groups.

The reward update $U_R$ vanishes whenever a group is reward‑uniform, so only mixed groups contribute any gradient signal.

Three concrete consequences follow. (i) The reward‑usable budget is gated by $\pi(p)$ and $G$, and empirical floors (A.3) reveal task‑level heterogeneity. (ii) On a reward‑uniform group the operator $U_R$ is identically zero, while the prospective operator $H$ remains unconstrained. (iii) Only the reward direction incurs a $\pi(p)^{-1}$ cost, whereas the prospective direction’s cost lacks this factor.

Group‑normalized advantages weight the gradient of $p$ by a scalar $a_G(p)$ that stays bounded away from zero, so the expected reward update follows a re‑weighted utility rather than raw success probability.

An all‑failed group ($E_0$) forces the reward update $U_R$ to zero, but the prospective update $H$ can still be non‑zero because it does not depend on the reward signal.

Covariance Analysis

We bound the sample complexity of the gating mechanism and expose its asymptotic dependence on the effective gain.

Proposition 4 (gating cost) quantifies how many independent groups are required to achieve a relative mean‑squared error $\epsilon^2$ when the policy and teacher are fixed.

The proof observes that $U_R$ vanishes off the mask $M$, so $\mu_R = q_G(p)\,\mathbb{E}[U_R\mid M]$. Conditional Jensen yields $\mathbb{E}\|U_R\|^2 \ge \|\mu_R\|^2/q_G(p)$, which translates into $\operatorname{tr}\Sigma_R \ge (q_G(p)^{-1}-1)\|\mu_R\|^2$. Substituting $q_G(p)=G_{\pi(p)}(1+O(\pi(p)))$ gives the asymptotic form in (16). The expression for $n_H$ follows from the relative‑error criterion $\operatorname{tr}\operatorname{Cov}(H)=V_{\text{row}}/G_{\text{eff}}$ and the bound $V_{\text{row}}\le M_v$.

Remark A.7 interprets the ablation in §RQ.1. Under per‑trajectory hindsight, the hindsight mean $\mu_H = (1-p)\mu_P + p\mu_K$ heavily weights the PITFALL channel because $1-p$ is large at typical success rates, making PITFALL rows the majority.

Because the prospective gradient’s divergence is bounded by the smoothness constant $L$ via $\|\delta_{il}\|^2 \le 2L D_{\text{il}}$, the measured KNOWLEDGE divergence remains stable while the PITFALL divergence drops by 30–47 %. This supports using PITFALL‑only SRD as a compute‑efficient default, though the bound remains one‑sided and a small divergence can still carry task‑relevant signal.

Error Correction Principles

Notes the missing problem statement and clarifies the required rule.

Missing problem statement prevents identification of specific pitfalls.

Do not generate pitfalls without a concrete problem; ensure the problem description is provided before predicting common errors.

Given the problem “Compute the integral of sin(x) from 0 to $\pi$,” one can predict the pitfall of forgetting the antiderivative constant. Without the problem, no pitfall can be listed.

Mechanism Audit

Auditing the causal chain of SRD on reward‑uniform groups

The appendix audits the four‑link causal chain: completed trajectories expose information, a retrospective teacher structures hindsight, SRD distills that hindsight into anticipation, and the anticipation changes behaviour. Each subsection (F.1–F.5) isolates one link and reports a phenomenon that should disappear if the link were absent.

Across the 686 scanned groups, 47.2 % are reward‑uniform: 24.6 % all‑fail and 22.6 % all‑success. SRD is applied to every group, even those with zero reward variation.

The table presents data across two channels: PITFALL and KNOWLEDGE. It includes columns for the number of records, foresight tokens, foresight mentions, hindsight mentions, and hindsight only. The value .299 in the "hind. only" column for the PITFALL channel is bolded.

In 29.9 % of PITFALL records the hindsight mentions an interface element that the foresight prediction omitted. This lexical proxy measures a gap SRD is designed to close, though we do not claim the mentions are always correct or actionable.

F.1 examines a group where the group‑relative RL objective yields zero gradient; the pre‑registered rule of §F selected this group (code × all‑fail × early) from 109 candidates. The run uses the code‑only Qwen3.5‑2B configuration with dynamic sampling disabled, leaving 31 of 32 groups all‑fail and a mean rollout reward of 0.012.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers