PaperGym: Rubric-Centered Evolution for Research-Plan Generation

Yuhan Wang, Zhengxi Lu, Yuchen Yan, Kaitao Song, Wenqi Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen

PaperGym turns research papers into training environments by decoupling research questions from grading rubrics to enable rubric-centered RL.

How can we create a scalable reinforcement learning environment for research-plan generation when there is no single "correct" answer?

Research planning is the bottleneck of automated science, yet existing reinforcement learning environments for this task suffer from "criterion leakage," where the grading rubric can be inferred directly from the question, allowing models to earn rewards through simple paraphrasing. PaperGym solves this by synthesizing research questions from a paper's goal and background while deriving grading criteria exclusively from its method and experimental design. This structural decoupling reduces criterion leakage to 3.7%, compared to up to 34.1% in previous datasets. The framework uses a two-stage training schedule: first, it uses the rubric as privileged context for on-policy self-distillation to build a broad prior, then as a reward signal for Group Relative Policy Optimization (GRPO) to refine the policy. This approach consistently outperforms supervised fine-tuning and single-stage training across all model scales.

Paper Primer

The core mechanism is a rubric-centered evolution: the model first learns to generate plans by distilling knowledge from a rubric-conditioned teacher, then optimizes those plans against the same rubric using GRPO. This "widen-then-narrow" curriculum ensures the model first absorbs a broad prior of valid research strategies before converging on high-reward trajectories.

PaperGym significantly reduces criterion leakage compared to existing benchmarks.

Leakage rates on PaperGym-20k are 3.7%, versus 11.9% to 34.1% in existing datasets.

The two-stage training schedule consistently outperforms supervised fine-tuning and single-stage RL.

Five-benchmark average improvements of +5.6, +5.0, and +4.8 points across Qwen3-1.7B, 4B, and 8B models. The trained Qwen3-8B model achieves 73.48 on ResearchQA, surpassing the larger Kimi K2.6.

Why is supervised fine-tuning (SFT) insufficient for research-plan generation?

SFT forces the model to imitate a single reference plan, which collapses output diversity and fails to align the model with the broader principles of scientific rigor and methodological innovation required for high-quality research.

What is the advantage of using the rubric twice in the training pipeline?

The first stage (self-distillation) uses the rubric to provide dense, token-level guidance that builds a broad prior, while the second stage (GRPO) uses the rubric as a sparse reward to verify complete plans, combining the benefits of dense distributional guidance and outcome-driven optimization.

By treating research papers as structured training environments rather than static text, PaperGym enables models to internalize the logic of scientific planning, moving beyond simple imitation toward verifiable, rubric-aligned research generation.

The Challenge of Research Planning

We expose why research planning lacks a verifiable critic and how PaperGym supplies one.

Research planning is the decisive capability of AI scientists, yet it admits no verifiable answer, so reinforcement learning lacks the environment—tasks paired with a critic—it requires. Existing pipelines draw both the research question and the grading criteria from the same paper content, which lets models earn rewards simply by paraphrasing the question, inflating criterion leakage to 11.90%–34.10%. PaperGym resolves this by synthesizing the question only from the research goal and background while extracting the criteria solely from the method and experimental design, cutting leakage to 3.7%.

A research plan is a concise blueprint that enumerates hypotheses, the methods to test them, and the experiments that will validate the results.

The core obstacle is the lack of verifiable ground truth for research plans, which prevents reliable reinforcement‑learning feedback.

The PaperGym Framework

Method details the two‑stage rubric‑centered training pipeline and its core components.

Existing rubric pipelines suffer two key problems: (1) criteria leak from answer to question, and (2) scalar rewards discard the rich supervision a rubric provides.

The framework turns each research paper into a self‑contained training environment where the question defines the task and a rubric acts as the critic.

The evolution uses the same rubric twice: first as privileged context for a self‑distillation teacher, then as the reward signal for policy optimization.

Stage 1: The teacher receives the full rubric and generates a rollout prefix “Use a low‑power MCU …”.

The student, lacking the rubric, produces “Use a low‑power MCU …” and is penalized by the KL divergence to the teacher’s distribution.

Stage 2: The frozen grader evaluates the student’s full response against each rubric item, emitting binary verdicts (e.g., method = 1, experiment = 1, completeness = 1, …).

The GRPO objective combines the specialized verdicts (weight $\alpha$ = 0.7) with the general verdicts to form the final scalar reward.

The two‑stage use of the rubric lets the model first learn a broad prior (many valid continuations) and then fine‑tune toward the exact criteria that matter for the specific research problem.

OPSD trains a student policy to match a teacher that sees the rubric, using a KL‑style loss that respects the order of generated tokens.

How does OPSD differ from ordinary self‑distillation that uses the reference answer as teacher?

Ordinary distillation conditions the teacher on the single reference answer, so the teacher’s distribution collapses to that answer. OPSD conditions the teacher on the full rubric, which describes many acceptable continuations; the student therefore learns a broader prior rather than mimicking one fixed solution.

GRPO treats rubric‑based binary verdicts as rewards and normalizes them across a batch of candidates, then optimizes the policy with a clipped surrogate objective and a KL penalty.

Why not use a learned scalar reward model instead of rubric‑based binary verdicts?

Binary rubric verdicts are directly interpretable and avoid the bias and instability of a learned scalar model, which would require its own training data and could misalign with the multi‑criterion evaluation the rubric already provides.

Extract plain‑text LaTeX from arXiv papers.

Split each paper into four logical stages (Goal, Background, Method, Experimental Design) using a map step that calls Qwen3‑235B‑A22B per section.

Group extractions by stage and merge duplicates in a reduce step, yielding a coherent four‑stage summary per paper.

Synthesize the research question from Goal + Background; synthesize the reference answer from Method + Experimental Design.

Generate specialized rubrics `R_spec` by prompting DeepSeek‑V4‑Flash for m = n = 10 criteria, then merge, deduplicate, and rank them.

Adopt the fixed general rubric `R_gen` (completeness, specificity, soundness, efficiency, safety) from prior work.

**Figure 1.** Overview of the PaperGym framework. (a) Data Generation: arXiv papers are parsed into four stages to synthesize questions and answers; rubrics are generated, merged, ranked, and filtered. (b) Rubrics-Based Policy Training: rubric-based OPSD followed by rubric-as-rewards GRPO.

The two‑stage rubric‑centered evolution gives a dense, criterion‑aware learning signal without requiring external reward models.

Empirical Results and Scaling

OPSD + GRPO boosts average scores by up to +5.56 across Qwen3 model scales.

OPSD + GRPO achieves the highest average scores across all Qwen3 model scales.

Average scores: 35.86 (1.7B), 35.84 (4B), 35.86 (8B).

**Figure 2.** Preliminary analysis. Left: Category breakdown of PaperGym-20k across three domains. Middle: Mean score vs. across-round range for four scoring models over five independent runs. Right: Pairwise inter-model agreement on binary rubric verdicts.

**Figure 3.** Left: Data scaling law with GRPO training. Right: Training dynamics across two-stage orderings of OPSD and GRPO on Qwen3-1.7B.

**Figure 4.** Entropy dynamics under different training strategies.

**Figure 5** Training dynamics across the two stage orderings of OPSD and GRPO on Qwen3-4B, the 4B counterpart of the right panel of Figure 3. OPSD→GRPO leads throughout training and converges to a higher final accuracy than GRPO→OPSD.

**Table 1.** Main results across model scales and training strategies.

**Table 4.** Ablation on data construction pipeline.

Performance gains grow with model size, and the OPSD → GRPO schedule consistently outperforms all alternatives.

Related Work

Appendix A details dataset analyses and ablations of the OPSD and GRPO training stages.

We first examine how the generated rubrics split between methodological and experimental criteria, then assess how much of each rubric can be inferred directly from the research question.

The table compares different types of "Privileged Information" across two categories: "Innov" and "Design". The rows represent three methods: "Rubric (Ours)", "Reference Answer", and "Rubric + Reference Answer". The values for "Rubric (Ours)" are bolded in both columns, indicating the highest performance.

Next we quantify criterion leakage by asking a frozen LLM whether each rubric item is directly inferable from the question alone, reporting leakage rates across in‑domain and out‑of‑domain benchmarks.

**Table 6.** Sensitivity of the GRPO stage to the reward mixing ratio (specialized : general). Best results are in **bold**.

We then compare the two possible stage orderings—OPSD→GRPO (default) versus GRPO→OPSD—across three model scales and five benchmarks.

**Table 7.** Benchmark performance under the two stage orderings of the two-stage schedule. $OPSD arrow GRPO$ is the default ordering; $GRPO arrow OPSD$ swaps the two stages with all other settings identical. Subscripts on the $GRPO arrow OPSD$ row denote the drop relative to $OPSD arrow GRPO$. The default ordering wins on every benchmark at every scale.

During GRPO training, each candidate response is scored by a frozen verifier model on both a specialized rubric and a set of seven general criteria; the final reward is a weighted sum 0.7 `R_spec` + 0.3 `R_gen` ($\alpha$ = 0.7).

Entropy analysis shows that OPSD steadily raises entropy, injecting diverse knowledge, while GRPO sharply reduces entropy, focusing the policy; the entropy drop is larger when OPSD precedes GRPO, confirming the benefit of a broaden‑then‑narrow curriculum.

Evaluation Protocol and Ablations

We evaluate how each training component affects pairwise win‑rate performance on the ResearchPlanGen‑ML benchmark.

Research planning lacks verifiable ground truth, so the paper extracts rubrics from existing papers to act as a critic for reinforcement‑learning training.

For each research question the judge looks at two generated plans and picks the better one (or declares a tie) on each rubric criterion; the win‑rate of a model is the fraction of questions where it wins.

**Figure 6.** Pairwise win rates of fine-tuned Qwen3-1.7B variants on ResearchPlanGen-ML. Models trained on our data outperform the RubricHub-trained counterpart, and both trained models defeat the untrained base model, evidencing benchmark quality gains and genuine training improvements.

Rubric Generation Guidelines

Defines strict constraints for constructing the final evaluation rubrics.

This section defines the constraints for constructing the final evaluation rubrics. Rubrics must focus on core innovation and key methodological steps, avoid trivial or peripheral aspects, and be limited to exactly ten items expressed as valid JSON wrapped in a ```json``` code block. Each rubric must be binary True/False, atomic, objective, positively phrased, and must not include disallowed fields or formatting.

Implementation Details

Appendix C enumerates the supplementary methodology, datasets, baselines, metrics, implementation details, and analysis strategies.

This appendix collects the full experimental setup: how the methods were applied, the datasets used, baseline comparisons, evaluation metrics, implementation specifics, and the analysis strategies employed.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers