RISE: Recursive Improvement via Self-Extrapolating Policy Distillation

Yang Li, Semih Yavuz, Shafiq Joty

RISE constructs a self-improving teacher by extrapolating the model's own RLVR training trajectory.

How can we improve language model post-training by creating a better "teacher" policy without relying on external models?

Reinforcement learning from verifiable rewards (RLVR) provides sparse, sequence-level feedback that fails to distinguish helpful reasoning steps from harmful ones, creating a credit-assignment bottleneck. RISE (Recursive Improvement via Self-Extrapolating Policy Distillation) resolves this by constructing a synthetic teacher through linear extrapolation of the model's own recent training updates, then distilling this "future" policy's dense token-level distribution back into the student. This recursive loop consistently outperforms RLVR-only training and privileged-conditioning baselines across math, STEM, code, and agentic tasks, with the largest gains on challenging competition benchmarks.

Paper Primer

RISE treats the model's training trajectory as a low-dimensional, approximately linear path in parameter or logit space. By projecting forward along the displacement between the current checkpoint and a trailing anchor, the method synthesizes a "future" teacher that represents a more capable version of the model, providing dense per-token supervision without requiring external models or privileged context.

RISE significantly improves performance on complex reasoning benchmarks compared to standard RLVR.

On OLMo3-7B, RISE (logit) improved AIME’24 accuracy from 30.2% to 46.9% (+16.7) and Math Avg from 47.6% to 56.4% (+8.8).

RISE enables faster convergence and higher sample efficiency than baseline RLVR methods.

Evaluation accuracy curves show RISE reaching higher accuracy earlier in training, with the gap widest in early stages before RL advantage estimates stabilize. Consistent outperformance at 1.3–1.6× wall-time overhead.

Why is this approach superior to simply using a stronger external model as a teacher?

External teachers suffer from distribution mismatch, as they are often conditioned on prefixes the student generates that the teacher never saw during its own training. RISE avoids this by using the student's own trajectory, ensuring the teacher remains on-distribution.

What prevents the extrapolated teacher from becoming noisy or degenerate?

The method uses a decaying extrapolation scale ($\beta$) that shrinks as training progresses, keeping the teacher within a safe trust region. Additionally, the RLVR phase grounds the displacement in verified reward, ensuring the extrapolation direction points toward genuine improvement.

RISE demonstrates that a model's own training history contains sufficient structure to serve as a self-improving teacher, effectively converting sparse outcome rewards into dense, fine-grained supervision.

Introduction and Motivation

We expose the teacher‑quality bottleneck in on‑policy distillation and outline our RISE solution.

On‑policy distillation (OPD) supplies dense, per‑token supervision for language‑model post‑training, yet its effectiveness stalls because the teacher is imperfect. External teachers drift from the model’s data distribution, and self‑distillation that relies on privileged conditioning cannot fully capture token‑level correctness due to limited in‑context learning capacity.

The core obstacle in OPD is that a noisy or mismatched teacher cannot provide reliable token‑level guidance, so the dense supervision it offers fails to improve the student.

RISE (Recursive Improvement via Self‑Extrapolating Policy Distillation) addresses this bottleneck by synthesizing a teacher from the model’s own RLVR trajectory: it extrapolates the recent displacement between a current checkpoint and a trailing anchor, turning a sparse outcome‑level update into a dense token‑level target without any external model or privileged context.

The bottleneck of teacher quality in OPD limits progress; RISE removes this barrier by generating a synthetic teacher from the model’s own trajectory.

Foundations: RLVR and OPD

We review RLVR, on‑policy distillation, and linear‑trajectory insights that motivate RISE.

This section situates our work among prior reinforcement‑learning and distillation approaches, highlighting why existing teacher constructions fall short of the needs identified in the introduction.

RLVR trains a language model to maximize a task‑specific reward that can be objectively checked, turning generation into a verifiable optimization problem.

OPD augments policy‑gradient training by forcing the student’s token‑level distribution to match that of a teacher model, providing dense supervision at every generation step.

GRPO computes a single advantage estimate per generated response and applies it uniformly to all tokens, simplifying variance reduction.

Task‑arithmetic work shows that fine‑tuning trajectories lie in low‑dimensional, nearly linear subspaces, enabling reliable extrapolation beyond the final checkpoint.

Because parameter updates concentrate in a few dominant directions, extending the line defined by the start and final checkpoints predicts a more capable future model.

Joint RLVR + OPD training combines sparse outcome rewards with dense teacher guidance, but all prior methods treat the teacher as fixed.

ExOPD uses a static external teacher, whereas RISE continuously updates a synthetic teacher by extrapolating the student’s own trajectory.

The RISE Mechanism

RISE builds a synthetic teacher by extrapolating the RLVR trajectory and distills it via OPD.

On‑policy distillation stalls because the teacher policy is limited to the latest RLVR update. RISE removes this ceiling by projecting beyond the current update, creating a stronger teacher without extra data.

RISE creates a synthetic teacher by extrapolating the model’s own RLVR‑grounded trajectory, then uses on‑policy distillation to transfer the richer token‑level signal to the student.

How does this extrapolation differ from simply taking a larger learning‑rate step?

A larger learning‑rate step still follows the same gradient direction but scales the update uniformly for all parameters. Extrapolation, by contrast, adds a *scaled displacement* after the RLVR update, preserving the RLVR‑induced improvement direction while allowing the teacher to lie beyond the region reachable by any single gradient step.

The mechanism adds a scaled displacement vector to the current policy representation, pushing it farther along the RLVR improvement direction.

Why not always use the cheaper logit‑space version?

Logit‑space extrapolation is cheap because it never creates a new model, but it assumes the network behaves linearly in its parameters. For highly non‑linear networks the approximation can diverge from the true weight‑space teacher, leading to poorer distillation quality.

Compute the displacement: $\Delta$$\theta$ = $\theta'$ₙ₊₁ − $\theta$ₙ = [1, 2].

Scale the displacement: $\beta$·$\Delta$$\theta$ = 2 × [1, 2] = [2, 4].

Add to the current parameters: $\theta_{f}$uture = $\theta$ₙ + $\beta$·$\Delta$$\theta$ = [1, 2] + [2, 4] = [3, 6].

Interpretation: the synthetic teacher’s parameters lie twice as far along the RLVR improvement direction as the RLVR step itself.

Extrapolation amplifies the RLVR‑induced improvement, creating a teacher that is stronger than the immediate RLVR update while still staying on the same improvement line.

RLVR phase: sample rollouts from $\pi$

Teacher construction: extrapolate either weight‑space or logit‑space using the current $\beta$ to obtain $\pi$

OPD phase: distill $\pi$

Anchor update: optionally smooth the anchor with an EMA (parameter $\eta$) before the next iteration.

Repeat for N iterations, gradually decaying $\beta$ toward 1.

Algorithm 1 – RISE (logit‑space)

Algorithm 2 – RISE (weight‑space)

**Figure 1.** **RISE overview.** (a) RLVR updates $\theta_n arrow \theta'_{n+1}$ (blue); extrapolation amplifies this displacement to construct $\theta_{\text{future}}$ (red); OPD distills $\theta_{\text{future}}$ into the student, yielding $\theta_{n+1}$ (green). (b) The training loop: RLVR grounds the direction, while OPD refines token-level decisions.

Empirical Results

RISE’s empirical gains, analysis, and ablations across math, STEM, code, and agentic tasks.

Recall that RISE builds a synthetic teacher by extrapolating the model’s own RLVR trajectory, then distills the resulting future policy back into the current student.

RISE (logit) improves AIME'24 accuracy from 30.2 % to 46.9 %, a +16.7 point gain.

30.2 % → 46.9 % on OLMo3‑7B (AIME'24); all baselines fall short of this jump.

**Figure C.1.** Per-metric convergence curves for all model configurations. Each row shows four representative benchmarks. RISE (both variants) consistently reaches higher accuracy in fewer training steps.

**Figure C.2.** Code generation convergence curves (Qwen3-8B-Base). Both RISE variants consistently converge faster than GRPO across all three benchmarks under both avg@4 and pass@4 metrics.

**Figure C.3:** **Grounding ablation: training reward and policy entropy.** Same three Qwen3-8B runs as Fig. 5a. Without RLVR (grey), training reward collapses to zero (left) while policy entropy spikes then crashes (right), confirming the phase-transition nature of the collapse.

**Figure 3.** RISE reaches higher accuracy on both math and STEM benchmarks.

**Figure 5.** Removing either phase hurts. (a) Without RLVR, accuracy collapses (solid) as generation length explodes (dotted). (b, c) Without OPD, training is stable but gains vanish.

**Figure 6.** Gain on AIME'24 (avg@16) from extrapolating Qwen3-1.7B GRPO checkpoints at varying $\beta$, relative to $\beta=1$. Shaded: RISE's operating range ($\beta_0=1.2$, decaying to 1).

**Figure 7.** Anchor ablation on Qwen3-1.7B. Lower $\eta$ improves Math Avg.

RISE consistently outperforms GRPO and standard OPD across scales and domains.

Theoretical Analysis

Key theoretical results clarify why extrapolating a teacher improves distillation.

The central premise—training large models via on‑policy distillation is limited by teacher quality, and RISE builds a synthetic teacher by extrapolating the RLVR‑grounded trajectory—reappears here as a theoretical lens.

For two policies $\pi_1,\pi_2$ generating responses of length $T$ with terminal reward $R(x,y)\in[0,1]$, $$ J(\pi_1)-J(\pi_2)=\sum_{t=1}^{T}\mathbb{E}_{s_t\sim\rho_t^{\pi_1}}\Big[\sum_{v}\big(\pi_1(v\mid s_t)-\pi_2(v\mid s_t)\big)\,Q^{\pi_2}(s_t,v)\Big], $$ where $s_t=(x,y_{<t})$ and $Q^{\pi_2}(s_t,v)=\mathbb{E}_{y_{t+1:T}\sim\pi_2}[R(x,y)\mid s_t,y_t=v]\in[0,1]$.

Start from the original performance‑difference lemma (Kakade & Langford, 2002) for discounted infinite‑horizon MDPs.

Identify states $s_t=(x,y_{<t})$ and actions $v$ (tokens) in the autoregressive setting.

Use $\sum_v\pi_1(v\mid s_t)=1$ to eliminate $V^{\pi_2}(s_t)$, obtaining $$ \sum_v\pi_1(v\mid s_t)A^{\pi_2}(s_t,v)=\sum_v\big(\pi_1(v\mid s_t)-\pi_2(v\mid s_t)\big)Q^{\pi_2}(s_t,v). $$

Insert the expression back into the sum over $t$ and recognize $Q^{\pi_2}\in[0,1]$.

Let $\pi_\theta$ be a student and $\pi_T$ a teacher. Then $$ J(\pi^*)-J(\pi_\theta)\le \underbrace{J(\pi^*)-J(\pi_T)}_{\text{teacher gap}} \;+\; T\,\underbrace{\overline{L}_{\text{OPD}}/2}_{\text{distillation error}}, $$ where $\overline{L}_{\text{OPD}}=\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}_{s_t\sim\rho_t^{\pi_\theta}}\big[D_{\mathrm{KL}}(\pi_\theta(\cdot\mid s_t)\,\|\,\pi_T(\cdot\mid s_t))\big]$.

Decompose the optimality gap as $J(\pi^*)-J(\pi_\theta)=[J(\pi^*)-J(\pi_T)]+[J(\pi_T)-J(\pi_\theta)]$.

Apply Lemma 1 with $\pi_1=\pi_\theta$, $\pi_2=\pi_T$ to bound the second term.

Since $Q^{\pi_T}\in[0,1]$, its range is at most $1$, allowing subtraction of a constant $c$ and application of Hölder’s inequality.

Use Pinsker’s inequality $\mathrm{TV}\le\sqrt{D_{\mathrm{KL}}/2}$ and Jensen’s inequality to obtain $$ J(\pi_T)-J(\pi_\theta)\le T\,\frac{\overline{L}_{\text{OPD}}}{2}. $$

Combine with the teacher‑gap term to reach the theorem statement.

Next we examine how extrapolating the teacher along a (approximately) linear trajectory influences the bound.

Assume $\phi(\pi_{\theta_n})=\phi(\pi_{\theta_0})+\alpha_n d$ with unit direction $d$ and $0=\alpha_0<\alpha_n<\alpha^*$. The extrapolated teacher $$ \phi(\pi_{\text{future}})=\phi(\pi_{\theta_0})+\beta\,\alpha_n d $$ is closer to $\pi^*$ than the current student iff $1<\beta<2\alpha^*/\alpha_n-1$.

Compute the distance from the extrapolated teacher to $\pi^*$: $\|\phi(\pi_{\text{future}})-\phi(\pi^*)\|=|\beta\alpha_n-\alpha^*|$.

The student’s distance is $\alpha^*-\alpha_n$.

Require $|\beta\alpha_n-\alpha^*|<\alpha^*-\alpha_n$, which simplifies to $1<\beta<2\alpha^*/\alpha_n-1$.

One RISE iteration that advances the trajectory coefficient from $\alpha_n$ to $\alpha'_{n+1}$ yields $$ J(\pi^*)-J(\pi_{\theta_{n+1}})\le \gamma(\beta)\,L\big(\alpha^*-\alpha'_{n+1}\big)+\frac{\overline{L}_{\text{OPD}}}{2}, $$ with contraction factor $\gamma(\beta)=\frac{\alpha^*-\beta\alpha'_{n+1}}{\alpha^*-\alpha'_{n+1}}<1$ whenever $1<\beta<2\alpha^*/\alpha'_{n+1}-1$.

Before OPD, the Lipschitz bound gives $J(\pi^*)-J(\pi_{\theta'})\le L(\alpha^*-\alpha'_{n+1})$.

The extrapolated teacher lies at $\phi$‑distance $|\alpha^*-\beta\alpha'_{n+1}|$, so $J(\pi^*)-J(\pi_{\text{future}})\le L|\alpha^*-\beta\alpha'_{n+1}|=\gamma(\beta)L(\alpha^*-\alpha'_{n+1})$.

Apply Theorem 2 with teacher $\pi_T=\pi_{\text{future}}$ and student $\pi_\theta=\pi_{\theta_{n+1}}$ (the OPD output) to add the distillation term $\overline{L}_{\text{OPD}}/2$.

Since $\gamma(\beta)<1$ exactly when $\beta$ lies in the safe range, the bound contracts.

We now quantify how low‑dimensional the actual training trajectories are, grounding the linear‑trajectory assumption.

**Figure A.1.** The RLVR weight trajectory is low-dimensional. Principal spectrum of the displacements $\delta_t = \theta_t - \theta_0$ from GRPO checkpoints on DAPOMath. Bars show variance explained per component; lines show the cumulative total. A perfectly linear trajectory would place 100% on the first component; the observed 68.2% reflects curvature along the path, while three components capture ~87% at both scales, confining the trajectory to a low-dimensional subspace.

Finally we discuss the bias introduced by the top‑$K$ approximation used when extrapolating logits.

With $K=100$, the top‑$K$ set captures > 99 % of the probability mass, and for the moderate extrapolation factors $\beta$ employed the mass shift to tokens outside $S$ is negligible, so the approximation error does not affect the theoretical guarantees.

Supplementary Results I

Ablation experiments quantify how each component of RISE contributes to performance.

This section isolates the impact of each RISE component by removing it and measuring the resulting performance change.

Removing the OPD phase leaves the in‑domain Math average essentially unchanged.

Table C.3 shows Math Avg moves from 60.0 % to 60.3 % at 8 B and from 45.4 % to 45.6 % at 1.7 B.

Extrapolation‑strength ($\beta$₀) and decay‑schedule ablations (Table C.4) were also evaluated, but the reported results show no statistically significant degradation compared with the default configuration.

Supplementary Results II

Ablation studies quantify the impact of resampling, anchor smoothing, and compute matching on RISE’s performance.

This section reports the ablations that isolate each component of the RISE pipeline, showing how performance degrades when a component is removed or altered.

Reusing rollouts instead of resampling hurts average accuracy by roughly 0.1 %.

Table C.5 shows the “Rollout reuse” variant achieves 84.8 % average versus 83.7 % for “Resample”.

Removing EMA smoothing (setting $\\eta = 1$) reduces logit‑space accuracy on OLMo3‑7B‑Instruct‑SFT by 6.3 points.

Table C.6 reports a drop from 82.9 % to 78.4 % (−6.3 logit) when $\\eta$ is changed from 0.1 to 1.0.

Even with an equal gradient budget, RISE (logit) outperforms GRPO‑2× by 8.2 % average accuracy.

Table C.7 lists RISE (logit) at 78.8 % versus GRPO‑2× at 70.6 % on the same benchmarks.

RISE’s Math‑Avg gains exceed the per‑method standard deviation by more than tenfold.

Table C.8 shows RISE improving Math Avg by +4.5 points while the reported standard deviations are ≤ 0.6.

Experimental Configuration

Detailed training, evaluation, and baseline configurations supporting the experimental comparison.

All experiments run on the VeRL framework on a single node equipped with 8 GPUs, each model trained for a single epoch over its respective dataset.

We use AdamW with a constant learning rate (no warmup), GRPO advantage estimation without standard‑deviation normalization, and token‑level importance‑sampling correction clipped at 2.0.

Prompts are truncated to 2,048 tokens and model responses to 8,192 tokens; all methods—GRPO, the three OPSD baselines, and both RISE variants—share the hyperparameters listed in Table B.1, isolating teacher construction as the sole source of performance variation.

Evaluation spans mathematical‑reasoning benchmarks (MATH‑500, AIME 2024/2025, AMC 2023, Minerva, OlympiadBench) and out‑of‑distribution checks (GPQA‑Diamond, IFEval, MMLU‑Pro), with additional multi‑domain suites (SuperGPQA, TheoremQA).

Code generation is measured on HumanEval+, MBPP+, and LiveCodeBench v6, while agentic tasks use ALFWorld (success rate) and WebShop (Score × 100 and Acc).

During evaluation we sample with temperature 1.0, top‑p = 1.0, and top‑k = −1 (unrestricted), mirroring the rollout distribution used in training.

Sample counts per prompt are 16 for AIME 2024/2025, GPQA‑Diamond, and Minerva; 8 for OlympiadBench; and 4 for MATH‑500 and AMC 2023. Deterministic benchmarks (IFEval, MMLU‑Pro, SuperGPQA, TheoremQA) use a single sample.

All Qwen3 models are evaluated in non‑thinking mode; code‑generation benchmarks use 4 samples each.

We report two complementary metrics: avg@N, the mean accuracy across $N$ sampled completions, and pass@N, the estimated probability that at least one of $N$ independent draws is correct.

The three OPSD baselines—GRPO+SDPO, SDAR, and RLSD—share a privileged teacher but differ in how its signal is incorporated into the update.

Privileged teacher construction proceeds by generating $n$ rollouts per prompt; if any rollout succeeds, the model is re‑prompted with that correct solution as privileged context, and its token distribution serves as the distillation target for all responses to that prompt.

If no rollout succeeds, only the RLVR loss is applied, yielding no distillation signal. The teacher is maintained as an exponential moving average of the student with a decay rate of 0.05.

Across training, the teacher is active on roughly 85 % of prompts for Qwen3‑8B and about 70 % for Qwen3‑1.7B.

GRPO+SDPO adds a KL‑divergence auxiliary loss toward the teacher’s token distribution; SDAR scales the GRPO advantage per token by the teacher–student probability gap; RLSD reweights advantage estimates using the teacher’s per‑token signal.

In contrast, RISE derives its teacher from the model’s own RLVR‑grounded training trajectory, providing a synthetic teacher uniformly for every prompt regardless of rollout success.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers