Questioning the Questions: Sustaining Self‑Evolution in Reasoning Models

Jinyuan Li, Chengsong Huang, Langlin Huang, Donghong Cai, Shiping Gao, Yuyi Yang, Jiaxin Huang

R-Quest sustains self-evolution by filtering invalid questions and suppressing repeated mathematical tasks.

Why does self-evolving reasoning performance collapse over time, and how can we sustain it?

Self-evolving reasoning models often suffer from performance collapse during repeated training rounds, as they inadvertently learn from poorly formed questions and redundant task variants. R-Quest addresses this by introducing two feedback signals: a validity-aware solver that rejects ill-posed questions, and a novelty gate that uses pairwise comparisons to identify and suppress repeated mathematical task structures. This approach sustains performance gains over ten rounds of self-evolution, outperforming previous methods by 17.32 points on mathematical benchmarks.

Paper Primer

The core issue is that self-generated training data degrades over time. Invalid questions—those with contradictions or missing information—become more frequent, and the model begins to overfit on repeated variants of the same mathematical tasks, which existing lexical-similarity controls fail to catch.

R-Quest is a co-evolutionary framework: it initializes the solver to explicitly recognize invalid questions, then uses this capability to provide validity feedback to the questioner. Simultaneously, it employs a frozen base model to perform sampled pairwise novelty checks, ensuring the questioner is rewarded only for generating novel mathematical task structures.

R-Quest prevents performance collapse and sustains improvement over ten rounds of self-evolution.

Comparison against the R-Zero baseline on Qwen3-4B-Base across 12 benchmarks in mathematics, code, and general reasoning. Outperforms R-Zero by 17.32 points in the final round of mathematical reasoning.

Why does the model's performance collapse in previous self-evolution frameworks?

The model accumulates invalid questions that lack unique solutions or contain contradictions, and it suffers from "diversity collapse" where it repeatedly generates variants of the same mathematical task that bypass simple lexical-similarity filters.

How does R-Quest avoid relying on stronger external models for feedback?

It uses a "validity-aware" solver initialized via mid-training to detect flaws, and a frozen copy of the base model to perform novelty judgments, keeping the entire feedback loop self-contained.

Sustained self-evolution requires moving beyond simple answer-consistency filtering; models must actively evaluate the structural validity and semantic novelty of their own generated training data.

Introduction and Motivation

We expose why self‑evolving reasoning models collapse and introduce R‑Quest to keep them improving.

Self‑evolving reasoning models iteratively train on their own generated questions, but this loop can erode performance over successive rounds. Two recurring quality problems—invalid questions and repeated variants of the same mathematical task—drive the collapse by feeding the solver low‑quality or redundant data.

**Figure 1.** Average performance on seven math benchmarks. R-Quest exceeds the peak of R-Zero in every round and peaks at round ten, leading by **17.32** points.

The performance ceiling in self‑evolution is caused by data quality degradation.

The R-Zero Baseline and Failure Modes

Identifying why self‑evolving reasoning collapses and what gaps remain.

Self‑evolving reasoning models repeatedly train on their own outputs, which quickly accumulate low‑quality, invalid, or redundant questions. This leads to a pronounced performance collapse as the solver sees increasingly noisy data.

R‑Zero is a self‑evolution loop where a questioner and a solver start from the same weights and alternate training, but it lacks any explicit filter for low‑quality or duplicated questions.

How does R‑Zero differ from a standard co‑training setup?

Standard co‑training typically assumes independent data streams for each model. R‑Zero ties the two models together by sharing initialization and feeding the solver’s own generated questions back into training, creating a closed feedback loop that can amplify errors.

The questioner proposes problems while the solver attempts to solve them; each iteration uses the other’s output as training data, forming a mutual learning loop.

Why does the BLEU‑based repetition penalty often miss duplicated question types?

BLEU measures surface‑level n‑gram overlap, so two questions that differ in wording or indexing (e.g., Question A vs Question B) can receive low similarity scores despite implementing the same underlying mathematical task. The penalty therefore underestimates true repetition.

**Figure 3.** Question validity rate and solver performance on Qwen3-4B-Base. (a) Valid-question rates before and after R-Zero’s answer-consistency filter. (b) Average math reasoning performance of the solver trained on matched original and repaired datasets.

**Figure 4.** Repeated question distribution and BLEU fragmentation in the fifth round of R-Zero. BLEU separates the illustrated mathematically equivalent questions into multiple different clusters.

R‑Zero collapses because it lacks a mechanism to distinguish between valid/invalid and novel/redundant questions.

The R-Quest Framework

Method combines validity and novelty feedback to keep self‑evolving models from collapsing.

Self‑evolving reasoning models tend to collapse when they repeatedly train on low‑quality or duplicated questions. R‑Quest counters this by filtering inputs with two orthogonal signals: validity feedback and novelty feedback.

**Figure 2.** Two question-quality problems in self-evolution and how R-Quest addresses them: (a) R-Quest can recognize and reject invalid questions; (b) R-Quest can detect repeated mathematical tasks despite differences in expression.

Validity Feedback teaches the solver to flag ill‑posed or unsolvable questions, so they never enter the training set.

Both questions are fed to the solver; it outputs $8$ for (i) and the token INVALID for (ii).

For (i) the indicator $1[a = a^{*}(q)]$ is 1, the second term is 0, so $R_{\text{init}} = 1$.

For (ii) the first indicator is 0, the second term is 1, so $R_{\text{init}} = -\lambda$.

The solver receives a positive reward for the correct answer and a penalty proportional to $\lambda$ for the wrongful rejection, shaping its future behavior.

Because the penalty only applies when the model says INVALID on a truly valid question, the solver learns to be cautious rather than overly aggressive in refusing.

How does Validity Feedback differ from simply discarding low‑confidence answers?

Low‑confidence filtering removes answers after the model has already attempted a solution, whereas Validity Feedback trains the model to *recognize* an ill‑posed question *before* committing to any answer, preventing the question from ever entering the training loop.

Novelty Feedback checks whether a newly generated question repeats a previously seen type, rejecting duplicates to preserve diversity.

For each $q_i$, we randomly pick two other questions without replacement.

Suppose $q_{3}$ happens to be paired with its duplicate; the gate rejects $q_{3}$.

All other questions avoid the duplicate in both sampled pairs, so they pass.

The empirical pass rate is $4/5 = 80\%$, matching the approximation $(1-0.2)^{2}=0.64$ (the discrepancy reflects randomness in small samples).

Even with a modest $K$, the sampler quickly filters out most duplicates while keeping the comparison budget linear in $B$.

Why not simply compare every pair of questions instead of sampling?

Full pairwise comparison guarantees that any duplicate is caught, but it scales quadratically ($O(B^{2})$) and becomes prohibitive for large batches. Sampling provides a controllable trade‑off: a linear $O(BK)$ cost with a tunable false‑negative rate that can be made acceptably low by increasing $K$.

Generate a batch of $B$ candidate questions with the questioner.

Run the validity‑aware solver on each candidate and collect the fraction $u(q)$ of INVALID responses.

Compute the validity gate $g_{\text{valid}}(q)=1-2\,u(q)$; discard any $q$ with $g_{\text{valid}}(q)<0$.

For each surviving $q$, sample $K$ other candidates and query the frozen base model for pairwise judgments $p_{ij}$.

Reject $q$ if any sampled $p_{ij}=1$; otherwise set $g_{\text{novel}}(q)=1$.

Sample multiple solutions for $q$, compute the largest answer cluster proportion $s(q)$, and form the uncertainty reward $R_{\text{uncertainty}}(q)=\min\{s(q),1-s(q)\}$.

Combine the gates into the final questioner reward $R_{\text{question}}(q)$ (Equation above) and update the questioner with GRPO.

Construct a training set from the accepted questions, mix in a small replay buffer of validity‑aware examples, and update the solver with GRPO.

Repeat the alternating updates for the desired number of self‑evolution steps.

Self‑evolution with validity and novelty feedback.

Experimental Results and Ablations

R-Quest filters low-quality questions with validity and novelty feedback, avoiding the performance collapse of self‑evolving models.

R-Quest outperforms R‑Zero by 17.32 points on average math performance after ten self‑evolution rounds.

Figure 5 shows the steady rise of R‑Quest to 51.92 % while R‑Zero drops to 34.60 %.

**Figure 5.** Average performance on seven benchmarks over ten rounds on Qwen3-4B-Base. Yellow and orange denote R-Quest without novelty or validity feedback.

**Figure 6.** Validity assessment, question and answer quality during self-evolution on Qwen3-4B-Base. Panels (a) and (b) show R-Quest's improved validity assessment on held-out questions. Panels (c) and (d) show that R-Quest maintains high question validity and better preserves majority-answer accuracy on valid questions. Bands in (c) show pointwise 95% Wilson confidence intervals.

**Figure 7.** Question type clusters (bars, left axis) and mathematical performance (lines, right axis) on Qwen3-4B-Base. Numbers above the bars denote top-five group proportion.

**Table 1.** Training Settings.

**Table.** Performance comparison of different methods on code generation and general-domain reasoning tasks for Qwen3-4B-Base and OctoThinker-3B models.

R‑Quest consistently outperforms R‑Zero, peaking at round 10 with a 17.32‑point lead.

Related Work

We review prior self‑improving reasoning work and highlight how validity and novelty feedback differ.

Label‑free reinforcement learning leverages model confidence, answer agreement, and entropy as signals (Zhao 2026b; Zuo 2026; Agarwal 2026). Complementary approaches such as Absolute Zero validate tasks via code execution (Zhao 2026a), while R‑Zero (Huang 2025) and Self‑Questioning Language Models (Chen 2026) train generators and solvers with model‑derived feedback. Broader frameworks embed learning in games, document corpora, vision tasks, and tool environments (Liu 2026; Liu 2025; Yue 2026; Huang 2026c; Li 2026b; He 2025; Dai 2026; Sun 2026).

Prior work on question validity in synthetic math data (Liang 2026; Shen 2026; Yang 2026) often relies on external models such as VHG (Lai 2026) with GPT‑5.4 verification. Our approach rejects invalid questions mid‑training, avoiding external calls thereafter. For novelty, methods like R‑Diverse (Lee 2026; Li 2026a) depend on a code generator (Qwen2.5‑Coder‑7B) and a code encoder (Jina‑Code‑Embeddings‑1.5B), whereas we assess task‑level novelty with a frozen copy of the base model using only a few random within‑batch comparisons.

Implementation Details and Robustness

Appendices provide dataset, training, and evaluation details for the validity‑aware mid‑training setup.

**Table 4.** Dataset statistics for validity-aware solver initialization.

**Table 5.** Error types among the 1,317 invalid questions in the training

**Table 6.** Training settings for validity-aware solver initialization.

**Table 7.** Question evaluation settings.

**Table 8.** GRPO training settings.

**Table 9.** Per-benchmark results for the gate ablations on Qwen3-4B-Base.

**Table 3.** Component ablations on Qwen3-4B-Base. Scores are averaged within each domain.

**Figure 8.** Approximate pass probability $(1 - p)^K$ for the default comparison budget $K = 8$ and the additional setting $K = 16$.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers