DiagEvo: Diagnosis-Guided Self-Evolution via Hierarchical Error Memory

Xincheng Wei, Yifan Ding, Yoshua Li, Dongsheng Ma, Rongxiang Weng, Xunliang Cai, Wenjian Ding, Yao Zhang

DiagEvo uses a hierarchical memory of recurring solver failures to guide question generation in self-play without external data.

How can we use a hierarchical memory of error causes to guide LLM self-play and prevent performance plateaus?

Self-play models often plateau because they lack a targeted curriculum, and existing solutions rely on external human examples or documents to steer question generation. DiagEvo instead extracts recurring error causes from the solver's own failed trajectories, storing them in a hierarchical memory that tracks whether a skill is "Active" or "Mastered." The challenger uses this memory to balance targeted practice on unresolved weaknesses with free exploration, while double-confidence filtering ensures the solver only trains on reliable, intermediate-difficulty questions. On Qwen3-8B, this approach reaches 72.3% mean accuracy on mathematical reasoning, outperforming R-Zero by 4.5 percentage points without using any external task resources.

Paper Primer

The core mechanism is a diagnostic loop: a lightweight model identifies the earliest reasoning difference between a solver's correct and incorrect responses, categorizing these as "error causes." These causes are organized into a hierarchy where "Active" causes trigger targeted question generation, and "Mastered" causes are used for cross-state stitching to maintain skill breadth.

DiagEvo achieves superior performance across all tested solvers (Qwen3-4B, Qwen3-8B, OctoThinker-8B) compared to both label-free and externally-guided baselines.

Mean accuracy across nine reasoning benchmarks. On Qwen3-8B, it reaches 72.3% on mathematical reasoning, exceeding R-Zero by 4.5 points and DARC by 1.2 points.

Double-confidence filtering significantly improves pseudo-label reliability by rejecting high-conflict questions.

Ablation studies comparing absolute-only filtering versus double-confidence (absolute + relative) filtering. Adding the relative confidence constraint improved the mathematical average by 1.4 points on Qwen3-8B.

Why is the solver's failure history a better curriculum signal than simply adjusting question difficulty?

Difficulty-based methods only track whether a question is hard, not why the solver fails. By diagnosing recurring error causes, DiagEvo identifies specific unresolved weaknesses, allowing the challenger to generate questions that target those exact gaps rather than just increasing length or complexity.

What is the role of the "Mastered" state in the hierarchical memory?

Mastered causes represent skills the solver has achieved high self-consistency on. They remain in memory to be used in "cross-state stitching," where they are paired with Active causes to provide context or maintain skill breadth during targeted generation.

Introduction

We expose why unguided self‑play stalls and how DiagEvo’s diagnosis‑guided curriculum revives progress.

Self‑play can drive language‑model self‑evolution, but without guidance the solver’s performance often plateaus or even declines as rounds progress. Unguided methods rely on generic signals such as difficulty, learnability, or diversity, which do not tell the challenger which unresolved reasoning weaknesses to target. Consequently, questions may grow longer and harder while the solver receives little targeted practice, leading to stagnation.

DiagEvo solves this by diagnosing solver failures and building a hierarchical error‑cause memory that guides curriculum generation. The diagnostician extracts recurring error causes, marks them Active or Mastered based on self‑consistency, and the challenger balances cause‑targeted generation with free exploration using these states. Double‑confidence filtering further discards ambiguous questions, ensuring the solver trains on intermediate‑difficulty items with a clear majority answer.

**Figure 1.** Co-evolution under pure free exploration and DiagEvo. (a) Pure free exploration produces much longer questions while solver performance stops improving. DiagEvo keeps question length stable and improves the mathematical average. (b) Example questions show how unguided self-play increases surface complexity across rounds. (c) DiagEvo uses a hierarchical error-cause memory to guide question generation across rounds.

The key shift is moving from unguided self‑play to a diagnosis‑guided curriculum that targets solver weaknesses directly.

Related Work

Related work surveys self‑play, curriculum, and noise‑reduction methods.

Self‑play drives LLM improvement by iterating a challenger that poses questions and a solver that answers them. Early label‑free baselines such as R‑Zero and Absolute Zero update both models without human annotations, while guided variants like R‑Few, SPICE, and DARC inject external resources. DiagEvo stays label‑free, using the solver’s failure history to steer question generation.

R‑Zero treats the solver’s own predictions as pseudo‑labels, updating both challenger and solver without any human‑written data.

DARC augments label‑free self‑play with difficulty labels, external documents, and a privileged teacher to improve pseudo‑label reliability.

Historical experience methods store past interactions to inform future decisions; Reflexion and ExpeL keep summaries of earlier dialogues, while CER, ReasoningBank, and ReMe build memories from both successes and failures. SkillRL converts trajectories into a reusable skill library, and curriculum techniques such as SvS and RLVR manipulate data diversity during reinforcement learning.

Unsupervised reinforcement learning suffers from majority‑voting bias, where repeated solver errors dominate pseudo‑labels. Prior work mitigates this with vote‑share thresholds or absolute confidence constraints, but they ignore the gap to the second‑most answer. DiagEvo introduces a relative confidence constraint, retaining only questions where the top answer clearly outpaces the runner‑up.

Label‑free self‑play baseline that updates challenger and solver using the solver’s own predictions as pseudo‑labels.

Guided self‑play method that combines difficulty labels, external documents, and a privileged teacher to improve pseudo‑label quality.

Label‑free self‑play system that diagnoses solver failures, builds a hierarchical error‑cause memory, and uses it to generate a curriculum of targeted questions.

Memory‑augmented language agent that stores summaries of past interactions to refine future decisions.

Transforms trajectories into a reusable skill library, updating it during policy training for more efficient reinforcement learning.

The DiagEvo Framework

We describe how DiagEvo uses diagnosed errors to drive targeted question generation and robust solver training.

Self‑play often wastes compute on questions the solver already masters or on questions that are too hard to learn from, slowing overall progress.

The system keeps a two‑level store: a set of skill nodes and, under each node, error causes that are either currently active or already mastered.

Initialize $V_1 = \{v_{\text{alg}}\}$ with description “basic algebra”.

Create $E_1 = \{e_1, e_2\}$ where $e_1$ = “off‑by‑one arithmetic”, $e_2$ = “mis‑interpreted variable name”.

Assign $a_1(e_1)=v_{\text{alg}}$, $a_1(e_2)=v_{\text{alg}}$.

Set $\text{st}(e_1)=\text{Active}$, $\text{st}(e_2)=\text{Active}$, and $f_1(e_1)=f_1(e_2)=1$.

Round 2 a new failure “division by zero” is extracted; a new skill node $v_{\text{calc}}$ is created and the cause is attached as Active.

The hierarchy lets the challenger sample from a focused subset of causes (e.g., only “basic algebra”) while still preserving the ability to introduce entirely new skill nodes when novel failures appear.

How does this hierarchical memory differ from simply keeping a flat list of error causes?

The hierarchy groups causes under skill nodes, which lets the challenger bias generation toward related families of mistakes and reuse higher‑level knowledge (the skill node) across many low‑level causes.

Using the memory, the challenger mixes two strategies: free exploration and cause‑targeted generation, with the mix gradually shifting toward the causes that have produced many failures.

Compute $z_3 = F_3 / F_1 = 6 / 12 = 0.5$.

Free‑exploration probability $\varepsilon_{4} = 12 / (12 + 6/2) = 12 / 15 = 0.8$.

Scaling factor \$1/(1+z_3/k) = 1/(1+0.5/2) = 1/1.25 = 0.8$.

Cause probabilities: $p_{4}(e_1) = 0.8 * (4/6) \approx 0.53$, $p_{4}(e_2) = 0.8 * (2/6) \approx 0.27$.

Thus the challenger samples a free‑exploration question with 80 % chance and, otherwise, picks $e_1$ about twice as often as $e_2$.

The schedule automatically adapts: as failures concentrate on a few causes, the challenger focuses on them; when failures become scarce, it re‑opens exploration.

How is this mixed‑generation schedule different from simply sampling causes uniformly?

Uniform sampling would ignore the observed failure frequencies, so the challenger would keep asking about rare or already‑mastered causes. The schedule weights causes by $f_t(e)$ and scales with $z_t$, ensuring the curriculum concentrates on the most problematic active errors while still preserving some exploration.

Before training the solver we discard questions that are either too easy (high consensus) or too ambiguous (conflicting top answers), keeping only those with moderate self‑consistency.

Compute $p_1(x)=3/5=0.6$ and $p_2(x)=2/5=0.4$.

Check absolute window: $0.4 \le 0.6 \le 0.8$ → passes.

Check relative constraint: $0.6 \ge 1.5 \times 0.4 = 0.6$ → passes (equality allowed).

Thus $x$ is retained for solver training.

If another candidate had votes (2,2,1) → $p_1=0.4$, $p_2=0.4$ → fails relative constraint (0.4 < 1.5·0.4).

The filter automatically discards questions that are either too ambiguous (near‑tie) or too certain (near‑full agreement), focusing training on the “Goldilocks” difficulty zone.

Why not just use a single confidence threshold instead of both absolute and relative constraints?

A single threshold cannot distinguish between a question that is uniformly easy (all responses agree) and one that is uniformly hard (responses are scattered). The absolute window removes both extremes, while the relative constraint catches subtle ties that would otherwise flip the pseudo‑label.

**Figure 1.** Overview of the proposed framework, which consists of four main components: (1) Challenger, (2) Solver, (3) Memory Maintenance, and (4) Hierarchical Error-Cause Memory.

Main Results

DiagEvo beats all baselines on every solver, setting new accuracy records.

DiagEvo outperforms every baseline on all three solvers, achieving the highest overall scores.

Table 1 shows DiagEvo (Diag. 235B‑A22B) leads each solver block, surpassing R‑Zero by 4.5 pts (math) and 2.6 pts (general) on Qwen3‑8B, and beating DARC by 1.2 pts (math) and 1.0 pts (general).

Scaling the diagnostician yields modest but consistent improvements: moving from 4B to 235B‑A22B adds roughly +1.0 pt to the mathematical average on each solver, with the largest gains on the hardest benchmarks such as OlympiadBench (+2.4 pts).

General‑reasoning performance is far less sensitive to diagnostician size; the general average changes by only 0.2–0.5 pts, and many general benchmarks show no gain.

Cross‑domain transfer is evident: training the solver only on mathematical questions still improves general‑reasoning benchmarks, raising the general average from 33.2 % to 38.8 % on Qwen3‑8B, from 27.8 % to 35.0 % on Qwen3‑4B, and from 10.7 % to 28.4 % on OctoThinker‑8B.

DiagEvo consistently outperforms baselines across reasoning benchmarks.

Ablation Studies

We ablate DiagEvo components on Qwen3‑8B‑Base to quantify each part’s contribution.

We evaluate three DiagEvo components—question generation policy, memory‑state updates with cross‑state stitching, and double‑confidence filtering—by ablating each on Qwen3‑8B‑Base. The mathematical average varies by at most 1.0 point across tested $k$ values (see margin).

**Table 2.** Ablation results on Qwen3-8B-Base. Each variant changes one component and keeps the remaining settings fixed. DiagEvo (full) is the shared reference for all three groups. It uses $\tau = 1.6$ for double-confidence filtering.

Freezing the challenger hurts performance.

Math Avg drops from 72.3 to 68.5.

Cross‑state stitching improves Math Avg over random‑pair stitching.

Math Avg is 1.0 point higher (71.3 vs 70.3) when stitching within the same skill node.

Adding the relative confidence constraint further raises Math Avg.

Math Avg increases by 1.4 points when the relative confidence constraint is added.

Co-Evolution Dynamics

Performance rises through five rounds before plateauing.

Recall that DiagEvo steers the challenger by diagnosing solver failures and storing error causes in a hierarchical memory. This memory evolves as training proceeds, gradually shaping the curriculum.

DiagEvo improves steadily for five rounds, attaining a 72.3% mathematical reasoning average at round 5.

Performance rises each round and peaks at 72.3% before a slight decline in later rounds.

The error‑cause memory expands from 151 to 244 distinct causes, yet end‑of‑round consolidation keeps the hierarchy compact. Skill nodes settle around 36 after round 3, providing a stable backbone for curriculum scheduling.

Active causes surge to 188 in round 2 then decline to 117 by round 5, while Mastered causes climb to 127, comprising more than half of the memory. Promotion resets closed‑episode frequencies, removing past failures and shifting sampling back toward free exploration.

Oracle agreement—a proxy for pseudo‑label quality—drops from 83% to 65% under pure free exploration with only an absolute confidence constraint. Introducing the memory raises this to 72% at round 5, and adding the relative confidence constraint pushes it further to 75%, yielding gains of 7 and 3 points respectively.

**Figure 3.** Co-evolution dynamics across training rounds on Qwen3-8B-Base. (a) Mathematical reasoning average after each round. Both curves use double-confidence filtering. (b) Numbers of Active causes, Mastered causes, and skill nodes. (c) Oracle agreement across rounds. Oracle agreement is the percentage of pseudo-labels that match oracle answers. The first setting uses pure free exploration with only the absolute confidence constraint. The second adds the memory under the same constraint. The third also adds the relative confidence constraint. Appendix H gives the measurement protocols.

**Figure 4.** Properties of questions generated after each round on Qwen3-8B-Base. (a) Average length in words. (b) Lexical diversity over adjacent word pairs, reported as 2-gram diversity. (c) Difficulty measured by the error rate of a fixed Qwen3-8B-Base solver against oracle answers. Both variants use double-confidence filtering. Pure free exploration removes the error-cause memory.

**Figure 5.** Principal component analysis (PCA) projections of questions generated in rounds 3–5. Colors indicate the generation mode.

Performance improves steadily across rounds as the memory matures.

Limitations and Generation

This section outlines current limits of DiagEvo and its mixed generation distribution.

Double‑confidence filtering keeps questions whose majority‑vote share falls within a preset range and whose top answer clearly outranks the runner‑up. Because the filter measures solver agreement rather than ground‑truth correctness, a shared error can still achieve strong agreement and pass the filter.

DiagEvo pre‑sets the number of co‑evolution rounds, using its memory only to pick questions inside each round. In the seven‑round analysis the average performance peaks at round 5, suggesting that a dynamic stopping criterion based on active causes or solver self‑consistency could decide when an extra round is worthwhile.

The current curriculum is built exclusively from mathematical questions, and gains on general‑reasoning benchmarks demonstrate transfer but do not evaluate direct curriculum construction in other domains. Extending the diagnosis‑memory loop to open‑ended tasks with long interaction sequences, such as multi‑turn tool use, is a promising direction for future work.

Challenger Update

How the challenger is refreshed each round and how its reward shapes the curriculum.

Each training round first generates a batch of fresh questions, uses the frozen solver to evaluate them, updates the challenger with GRPO, and then draws a new pool of curriculum candidates for the next solver‑training batch.

The challenger is rewarded for proposing questions that sit at the solver’s competence boundary (≈ 50 % self‑consistency) while avoiding redundant queries that form large clusters.

Compute uncertainty rewards: $r_{\text{unc}}(x_1)=0.5-2\cdot0.6=-0.7$, $r_{\text{unc}}(x_2)=0.5-2\cdot0.4=-0.3$, $r_{\text{unc}}(x_3)=0.5-2\cdot0.5=0$, $r_{\text{unc}}(x_4)=0$.

Derive distances: $d_{12}=0.2$, $d_{13}=0.7$, $d_{14}=0.7$, $d_{23}=0.8$, $d_{24}=0.8$, $d_{34}=0.1$.

Agglomerative clustering with the R‑Zero criterion merges the most similar pair first ( $x_3$ and $x_4$ since $d_{34}=0.1$ ), forming cluster $C_1=\{x_3,x_4\}$. The remaining points stay separate, yielding clusters $C_2=\{x_1\}$ and $C_3=\{x_2\}$.

Compute repetition penalties: $r_{\text{rep}}(x_3)=r_{\text{rep}}(x_4)=|C_1|/4=2/4=0.5$; $r_{\text{rep}}(x_1)=r_{\text{rep}}(x_2)=1/4=0.25$.

Final rewards: $r_{\text{chal}}(x_1)=\max\{0,-0.7-0.25\}=0$, $r_{\text{chal}}(x_2)=\max\{0,-0.3-0.25\}=0$, $r_{\text{chal}}(x_3)=\max\{0,0-0.5\}=0$, $r_{\text{chal}}(x_4)=0$. All four questions receive zero reward because either they are too easy or belong to a redundant cluster.

The example shows that the reward simultaneously filters out overly certain questions and penalizes clusters, ensuring the challenger only keeps questions that truly challenge the solver.

Sample a challenger batch $Q_{\text{chal}}^{t+1}\sim q_{t+1}(\cdot\mid M_t;\theta_t)$.

Freeze the current solver and obtain majority‑answer vote shares $p_1(x)$ for each $x\in Q_{\text{chal}}^{t+1}$.

Compute the challenger reward $r_{\text{chal}}(x)$ using equations (6)–(7) and update the challenger parameters $\theta_{t+1}$ with Group Relative Policy Optimization (GRPO).

With the updated challenger, sample a new pool of curriculum candidates $Q_{\text{cur}}^{t+1}\sim q_{t+1}(\cdot\mid M_t;\theta_{t+1})$.

Pass $Q_{\text{cur}}^{t+1}$ to the solver‑training pipeline (Section 3.2) to build the next solver training set.

How does this challenger reward differ from a standard reinforcement‑learning reward that simply maximizes solver error?

Standard RL rewards would push the challenger to generate the hardest possible questions, often leading to unanswerable or overly noisy queries. By contrast, the challenger reward explicitly balances difficulty (via $r_{\text{unc}}$) with diversity (via $r_{\text{rep}}$), so the curriculum stays solvable yet still drives learning progress.

State–Frequency Transition Rule

The transition rule upgrades causes based on self‑consistency and solver failures to guide curriculum.

Self‑play often wastes compute on questions that are either redundant or too hard. The method therefore needs a rule that promotes causes only when they show reliable self‑consistency, yet remains flexible enough to backtrack if the solver later fails.

When a cause repeatedly yields high self‑consistency, we tentatively promote it to Mastered, but we keep the option to roll back after observing actual solver failures.

How does this promotion differ from standard curriculum learning that orders questions by difficulty?

Standard curricula typically rank questions by an externally estimated difficulty and advance monotonically. Here promotion is driven by the cause’s own self‑consistency (acct) and is provisional: the cause can be promoted early if it appears solvable, but the later $\Delta$ₜ(e) count lets the system pull back if the Solver actually fails, providing a feedback loop absent in static difficulty schedules.

Compute acct for each cause: e₁ gets 0.80, e₂ gets 0.60.

Apply the promotion rule: e₁ satisfies all conditions (Active, |Pₜ|>0, acct≥0.75) → provisional state (Mastered, 0). e₂ fails the acct condition → keep (Active, 0).

Run the Solver; suppose it produces $\Delta$ₜ(e₁)=0 failed trajectories and $\Delta$ₜ(e₂)=1 failed trajectory.

Commit the final states: e₁ remains (Mastered, 0); e₂ stays (Active, 0) but the failure count will influence the next round’s promotion decision.

The rule promotes only causes that consistently exhibit high self‑consistency, while still allowing the Solver’s actual performance ($\Delta$ₜ) to correct premature promotions in subsequent rounds.

By coupling self‑consistency–driven promotion with a post‑solver failure check, the transition rule steers the Challenger toward questions that are challenging yet solvable, reducing wasted compute on redundant or impossible queries.

Hyperparameter Sensitivity

We examine how $k$ and $\tau$ affect performance, identifying the best settings.

The schedule depends on two hyperparameters: $k$, which trades off exploration versus exploitation, and $\tau$, which sets the confidence threshold for the double‑confidence filter.

Setting $k=0.5$ yields the highest overall performance.

Table 8 shows the peak Math Avg of 72.3 and General Avg of 38.8 at $k=0.5$, with drops of up to 0.9 points when $k$ deviates.

Table 9 varies $\tau$: a loose setting ($\tau=1.2$) gives 71.6/38.4, the best tested ($\tau=1.6$) recovers the full configuration with 72.3/38.8, and a strict setting ($\tau=2.0$) drops to 71.3/38.1, confirming that overly strict filtering removes too many training questions.

Setting $\tau=1.6$ gives the best results.

Table 9 peaks at $\tau=1.6$ with 72.3 Math Avg and 38.8 General Avg, while stricter filtering ($\tau=2.0$) reduces scores.

Case Study: From Failure to Curriculum

From a diagnosed failure to a targeted curriculum item, self‑consistency climbs to 75 %.

Self‑play models often waste compute on redundant or overly hard questions; DiagEvo diagnoses solver failures and steers the challenger to generate a curriculum of targeted, solvable‑but‑challenging questions.

Self‑consistency on the diagnosed cause rises from 48.2 % to 75.0 %, surpassing the 70 % promotion threshold.

Across three rounds the mean self‑consistency scores increase to 48.2 %, 65.6 %, and finally 75.0 %.

**Figure 6.** From failure to curriculum (Qwen3-8B-Base). (a) The diagnostician compares trajectories that agree and disagree with the pseudo-label and extracts a transferable error cause. (b) The cause enters the hierarchical memory as Active. (c) Cross-state stitching combines it with a Mastered cause into a composite question. (d) Rising self-consistency on later targeted questions promotes it to Mastered.

The improvement mirrors the paper’s central premise: diagnosing a specific failure and feeding it back as a curriculum item yields measurable gains without altering the overall training pipeline.

A concrete failure—missing a domain‑restriction check—is turned into a targeted question that drives the solver from 48 % to 75 % self‑consistency.

Training Hyperparameters

Key hyperparameter settings and runtime breakdown for reproducing DiagEvo.

The following tables enumerate the exact optimizer, batch, and curriculum parameters used to reproduce DiagEvo, followed by a concise runtime profile of the full training loop.

**Table 3.** GRPO optimization settings for the challenger and solver.

**Table 4.** DiagEvo-specific training configuration (challenger and solver co-evolution loop).

**Table 5.** Reward, filtering, curriculum, and memory-maintenance hyperparameters used throughout DiagEvo.

**Table 6.** Wall-clock share of each component in a complete DiagEvo run with the default configuration. Diagnosis and memory maintenance, the components specific to DiagEvo, account for 5.7% of the total time.

Baseline Validation

We verify baseline scores by reproducing them under a common evaluation protocol.

We re‑evaluate the three base models and reproduce the R‑Zero baseline using the same solvers, benchmarks, and evaluation protocol as the original studies.

Table 7 shows that base‑model averages match within 0.1 points and R‑Zero scores within 0.5 points on every column, with per‑benchmark deviations below 1.0 points.

**Table 7.** Aggregate scores of the reported baselines and of our reproduction under the same evaluation protocol. Base-model averages match within 0.1 points and R-Zero scores within 0.5 points on every column.

Diagnostic Evaluation Protocols

We detail the evaluation protocols and metrics underlying Figures 3‑5.

All experiments for Figures 3‑5 use the Qwen3‑8B‑Base model together with the default 4B diagnostician, and share the training configuration described in Appendix E.

Per‑round performance is evaluated after each training round using the protocol from Section 4.1, reporting the average mathematical reasoning score of the current solver.

In Figure 3(a) both variants keep double‑confidence filtering; the only difference is whether question generation consults the error‑cause memory.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers