Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development

Yiwei Li, Wanli Yang, Hexiang Tan, Xiangzhou Huang, Zhengyu Chen, Ziran Li, Borun Chen, Shanglin Lei, Huaisheng Zhu, Hao Tian, Fei Sun, Xunliang Cai, Jingang Wang

Current AI research agents function as engineering optimizers, not innovators, with performance limited by process bottlenecks and unstable experience reuse.

How do autonomous research agents actually perform across the full lifecycle of long-horizon R&D, beyond just their final success scores?

Autonomous research agents are evaluated using final scores that obscure why a model succeeds or fails, making it impossible to distinguish between effective planning, reliable execution, and the ability to learn from past mistakes. The authors decompose the research loop into three deterministic metrics—Solution Framing, Execution, and Feedback Control—and use controlled experiments to isolate how accumulated experience and harness design influence performance across 36 long-horizon tasks. While agents can implement practical solutions, they rarely produce methodological novelty; instead, they primarily recombine established techniques or exploit evaluation loopholes, with performance stability varying significantly across models.

Paper Primer

The research process is modeled as a closed-loop cycle: the agent proposes a direction, implements changes, observes verifier feedback, and decides how to proceed. The authors treat this as a diagnostic pipeline where Solution Framing (C1) measures the quality of the chosen direction, Execution (C2) measures the reliability of implementation, and Feedback Control (C3) measures the ability to retain progress and recover from regressions.

Outcome scores mask distinct process-level failures.

Models with similar final scores often show divergent strengths; for example, GPT-5.5 excels in Execution while Gemini-3.1-Pro is stronger in Feedback Control. The gap between the strongest and weakest models is 0.237 on average performance, but only 0.122 on best-case performance, indicating that reliability—not peak capability—is the primary differentiator.

Genuine methodological novelty is rare in current automated research.

In a manual review of 252 solutions, only 3 (1.2%) qualified as novel approaches, while 111 (44%) were compositions of established techniques and 16 (6.3%) were evaluation-specific shortcuts. Agents are over five times more likely to exploit loopholes in the evaluation protocol than to produce a validated novel approach.

Why is a final score insufficient for evaluating autonomous research agents?

A final score does not reveal whether a model reached a solution through early insight or extensive trial-and-error, nor does it indicate if the agent can reliably implement its ideas or learn from its own experimental history.

Does accumulated experience always improve agent performance?

No; while experience often helps agents avoid dead ends and preserve useful discoveries, it can also anchor them to local optima or carry forward misleading conclusions, leading to performance degradation in some cases.

Beyond Final Scores

We argue that evaluating autonomous agents requires process-aware metrics beyond final scores.

Current benchmarks evaluate autonomous research agents solely by their final outcome, which conceals where progress is made, how efficiently the process runs, and whether accumulated experience informs later decisions.

To understand autonomous agents we must look inside the research loop, measuring not just the end score but how the agent frames solutions, executes them, and controls feedback, plus whether past experience shapes future choices.

Agent A: Execution = 0.91, Feedback Control = 0.62.

Agent B: Execution = 0.63, Feedback Control = 0.89.

Both agents receive the same final score of 0.78, so a pure outcome metric would rank them equally.

Outcome‑only evaluation masks these internal capability differences, preventing diagnosis of where improvement is needed.

**Figure 1.** Analytical views used to interpret behavior in automated research. The process view covers Solution Framing (C1), Execution (C2), and Feedback Control (C3). The experience view uses controlled comparisons to measure how accumulated experience affects subsequent decisions in intra- and inter-task settings.

The shift from outcome‑only to process‑aware evaluation is essential for diagnosing and advancing autonomous research agents.

Outcome-Level Performance

Outcome-level evaluation reveals performance rankings and cost trade-offs across seven frontier models.

We evaluate seven frontier models on a shared harness to compare final performance and resource use.

Outcome-Level Evaluation measures the final quality of an agent’s solution, ignoring how the solution was reached.

Opus‑4.7 ranks first on both avg@3 and best@3.

Opus‑4.7 achieves $0.739$ avg@3 and $0.790$ best@3, outperforming the nearest competitor by $0.029$ avg@3.

**Figure 2.** Outcome-level performance across seven models. Solid segments indicate avg@3, while full bar heights indicate best@3. (a) Overall performance, ranked by avg@3. (b) Category-level performance; filled and open circles mark the avg@3 and best@3 leaders, respectively.

**Figure 3.** Mean estimated inference cost per task across four task categories and overall. Values average over the three independent rollouts for each model–task pair. For consistent cross-model comparison, all input tokens are priced without cache discounts.

**Table 1.** Per-category avg@3 (top) and best@3 (bottom) across seven models, ordered by overall avg@3. The bold entry marks the avg@3 and best@3 leader in each category.

Performance rankings and cost‑efficiency trade‑offs together determine the most practical model for a given budget.

Decomposing the Research Loop

Decompose the research loop into three measurable stages and describe how they are computed.

The research loop consists of repeated rounds where an agent proposes a direction, implements it, observes the outcome, and decides the next step. A single final score cannot reveal which part of this loop succeeded or failed, so we break it into three process dimensions.

We measure the research loop by three orthogonal scores—Solution Framing (C1), Execution (C2), and Feedback Control (C3)—each capturing a distinct failure mode.

How does C1 differ from the outcome-level evaluation used earlier?

C1 focuses on the *trajectory* of the best verifier score during the run, rewarding early progress and penalizing later regressions, whereas outcome-level evaluation only looks at the final score after the entire process.

Define three dimensions (C1, C2, C3) that map to the causal stages of the research loop.

At each evaluated checkpoint, record verifier feedback (e.g., best score, build success, regression flags).

Compute deterministic scores for each dimension from the recorded signals using the rules in Appendix C.

Average scores across valid random seeds for each model‑task pairing.

Weight tasks equally when aggregating model‑level results.

**Figure 4.** Process dimensions across seven models. All values are averaged over three rolls.

**Figure 5.** Process dimensions by task category, averaged over the seven models.

**Figure 6.** Behavioral diagnostics across seven models. Each cell reports the exact value, while darker shading indicates a larger value within the same column and does not imply stronger capability. Ratios are shown as percentages, scores as decimals, and counts as averages. The gray column reports the average number of evaluated commit rounds as observation support. Values are averaged across repeated runs for each model and task.

These process metrics expose bottlenecks that final outcomes hide, enabling targeted improvements to solution framing, execution reliability, or feedback control.

Learning from Experience

We assess how agents reuse experience within and across tasks, revealing varied gains.

To understand whether agents can leverage their own past work, we measure two complementary capabilities: improving the next commit on the same task (intra‑task) and transferring lessons to a new task (inter‑task).

Agents accumulate knowledge during a run and can reuse it later, either to refine the same problem or to help solve a different one.

**Figure 7.** Per-model first-commit score with and without retained experience (bars, left axis) and the corresponding intra-task gain $\Delta$ (line, right axis), averaged over 32 retained trajectories.

Intra‑task self‑improvement yields an average gain of +0.045 across models, with the strongest gain of +0.145 from LongCat‑2.0.

Aggregated over 32 tasks, every model except Kimi‑K2.7‑Code shows a positive $\Delta$; LongCat‑2.0’s $\Delta$ is the highest reported.

**Figure 8.** Per-model avg@3 with and without trajectory-derived experience (bars, left axis) and the corresponding inter-task gain (line, right axis).

Inter‑task transfer shows DeepSeek‑V4‑Pro achieving the largest avg@3 gain of +0.093, while GPT‑5.5 improves both avg@3 (+0.063) and best@3 (+0.022).

Across 19 held‑out targets, DeepSeek‑V4‑Pro’s $\Delta_{i}$nter on avg@3 is +0.093; GPT‑5.5’s $\Delta_{i}$nter is +0.063 on avg@3 and +0.022 on best@3. +0.093 max avg@3 gain, +0.063 avg@3 gain (GPT‑5.5), +0.022 best@3 gain (GPT‑5.5)

**Figure 15.** Task-level signs of self-improvement. Positive, tie, and negative denote the sign of the score difference between conditions with and without experience. (a) Intra-task outcomes over 32 tasks (b) Inter-task outcomes over 19 targets (avg@3)

**Figure 16.** Inter-task self-improvement under best@3. (a) Per-model best@3 with and without trajectory-derived experience (bars, left axis) and the corresponding $M_{best}$ (line, right axis), averaged over 19 targets. (b) Numbers of targets with positive, tied, and negative best@3 gains.

The Role of the Harness

How the choice of harness shapes agent performance and stability.

We first isolate the effect of the agent harness by swapping three harnesses while keeping tasks, environment, and rollout protocol identical.

The harness is the thin software layer that mediates an agent’s tool use, records its actions, and enforces high‑level research‑loop policies.

How does an “Agent Harness” differ from a simple script that calls tools?

A script merely executes commands; the harness also interprets the agent’s intent, validates inputs, records outcomes, and can enforce research‑loop policies such as version control or stability checks. Those extra responsibilities shape the agent’s behavior beyond raw tool use.

Beyond raw scores, harnesses support the research loop through tool‑failure recovery, context management, and explicit task‑tracking mechanisms.

**Figure 9.** Coding harness comparison.

Harness choice significantly impacts agent performance.

Analyzing Solution Novelty

We quantify how often agents produce truly novel solutions versus reusing known tricks.

The paper’s core premise is that evaluating agents only by final outcomes hides differences in process efficiency, learning ability, and solution novelty. This section examines how often agents actually generate novel ideas.

Only a tiny fraction of agents’ best‑of‑three solutions are genuinely novel.

3 out of 252 solutions were labeled as novel after manual review.

Solution Novelty classifies a discovered solution by whether it introduces a new idea beyond recombining existing techniques.

**Figure 11 |** Novelty analysis of 252 best-of-three solutions. Left: distribution across the eight solution categories after Opus-4.8 classification. Right: the three novel approaches retained after manual review.

High scores do not guarantee novel discovery.

Related Work

Survey of benchmarks and prior evaluations for autonomous research agents.

Autonomous research agents have been evaluated on a growing suite of benchmarks that stress code modification, experiment execution, and long‑horizon planning. Prior work also introduced process‑level metrics and experience‑reuse evaluations, highlighting gaps that our framework aims to fill.

Benchmarks language‑model agents on realistic code‑modification and experiment‑running tasks, measuring how well they improve ML systems under resource constraints.

Evaluates agents on end‑to‑end machine‑learning engineering tasks, requiring them to iteratively refine models and report improvements.

Extends earlier benchmarks by requiring agents to not only modify code but also to reason about experimental design under realistic constraints.

Focuses on post‑training adaptation where agents must improve already‑trained models without full retraining.

Evaluates long‑horizon agents that repeatedly propose modifications, observe feedback, and refine artifacts over many cycles.

A comprehensive suite that combines code modification, experiment execution, and resource‑bounded planning for autonomous research agents.

Analyzes aggregate search behaviors such as improvement frequency, exploration diversity, and search depth.

Focuses on functional‑module learning where agents must discover and integrate reusable components.

Introduces “progress rate” to quantify advancement toward intermediate subgoals in long‑horizon tasks.

Evaluates correctness of tool selection, argument formulation, and execution order in autonomous agents.

Tracks semantic environment states to separate exploration reach from execution accuracy.

Compares software‑engineering trajectories against successful process references to identify inefficient behaviors and “lucky passes.”

Converts task feedback into verbal reflections that guide subsequent attempts.

Extracts reusable insights from prior trajectories for cross‑task transfer.

Extends evaluation from isolated episodes to sequential task streams, measuring experience accumulation and skill transfer.

Measures experience accumulation, skill transfer, and longer‑term evolution of autonomous agents.

Studies whether procedural knowledge or trace‑derived abilities improve performance across tasks.

Evaluates evolutionary strategies for agents, focusing on how iterative improvements affect long‑horizon research tasks.

Studies scaling laws for learning from environments, closely related to intra‑task self‑improvement.

Shows that the design of the agent–computer interface (the harness) can substantially affect software‑engineering performance.

Jointly analyzes models, scaffolds, and benchmarks under standardized evaluation to isolate the impact of each component.

Directly measures harness effects across multiple model backends and execution configurations.

Argues that long‑horizon agent comparisons require explicit harness disclosure and controlled evaluation protocols.

Model-Specific Performance Nuances

This section details performance gaps, resource trade‑offs, and formal process metric definitions.

Kimi shows the widest avg–best gap (0.240), indicating that many models can find strong solutions but differ in reliably reaching them.

In System Optimization, Opus tops avg@3 (0.675) while Opus, GPT, and GLM have almost identical best@3 scores (≈0.70), highlighting consistency differences over peak performance.

Puzzle & Challenge exhibits the smallest separation: the max gap is 0.150 on avg@3 and 0.074 on best@3, with GLM narrowly leading both metrics.

CUDA shows the largest performance split, with gaps of 0.403 (avg@3) and 0.414 (best@3); Opus dominates avg@3, whereas GPT attains the highest best@3.

Beyond scores, we report wall‑clock time and token consumption for each category, providing a complete resource‑performance picture that complements the aggregated best@3 view.

Formal definitions for the process metrics are given, covering proposal gating, solution framing (C1), execution (C2), feedback control (C3), and aggregation across seeds and tasks.

C.1 aligns official score checkpoints with transcript commits, removing only administrative or non‑mutating checkpoints while preserving those that affect the score.

C.2 defines C1 as the high‑water‑mark recurrence $h_i = \max(h_{i-1}, x_i)$ and averages early, middle, and late stages over a horizon of 20 checkpoints.

C.3 introduces C2, where each checkpoint’s success is discounted by the schedule $d(n)$ (1.00, 0.85, 0.70, 0.60, 0.50) based on the number of build failures.

C.4 describes C3’s feedback control: dips are identified when scores drop by more than $\epsilon$ = 0.01, and recovery credit $ρ_e$ is computed from the dip depth and subsequent recovery.

C.5 aggregates metric scores by first averaging seeds per model‑task pair, then averaging across tasks to ensure task‑balanced results.

D summarizes the behavioral diagnostic components, tying together the defined metrics for comprehensive agent evaluation.

**Figure 12.** Resource–performance trade-offs across seven models. The panels compare best@3 against total estimated cost (left), mean wall-clock time per task (middle), and mean interaction steps per task (right).

**Figure 13.** Mean wall-clock time per task across four task categories and overall. Values average over the three independent rollouts for each model–task pair.

**Figure 14 |** Mean token consumption per task across four task categories and overall, including both input and output tokens. Values are reported in millions and average over the three independent rollouts for each model–task pair.

Intra-Task Experience Analysis

Ablation results reveal when retained experience aids or harms agents and how reuse form impacts transfer.

We first dissect intra‑task memory effects, separating cases where retained experience improves the next commit from those where it degrades performance.

Positive effects fall into three patterns: (1) avoiding a known dead end, (2) reusing a tuned configuration, and (3) reusing a hard‑won implementation.

Negative effects arise from two patterns: (1) carrying over a premature conclusion, and (2) anchoring to a local optimum.

**Figure 17.** How the representation and source of experience affect inter-task reuse. (a–b) M under explicit reuse of extracted lessons and implicit reuse of the raw source workspace. (c–d) M when the executing model uses self-generated lessons or lessons transferred from another model. In (c–d), the upper model produces the lessons and the lower model applies them.

Diagnostic Methodology

Appendix D details the diagnostic metrics used to evaluate long‑horizon R&D agents.

Figure 6 visualizes how the research loop progresses, but it does not contribute an extra capability score. Each run‑level value is first averaged across valid seeds per model and task, then the 36 tasks receive equal weight. The tolerance for a meaningful score change is set to $\epsilon$ = 0.01.

In the C1 Search Shape diagnostics, let $x_1,\dots,x_T$ be the scores of evaluated agent proposals; the repository baseline is excluded. We report the best observed score $p=\max_i x_i$, early capture $x_1/p$ (only when $p>0$), and later headroom capture $(p-x_1)/(1-x_1)$ if $x_1<1$, otherwise 0. Early capture measures initial solution quality, while later headroom capture quantifies progress after the first proposal.

The C2 Build Behavior metrics extract build commands from shell transcripts and align them to canonical proposals. For each matched proposal $i\in I$, $b_i$ counts recognized build invocations before the commit and $f_i$ counts those that cause a code‑related failure; the fraction of rounds with build errors is $|\{i\in I:f_i>0\}|/|I|$. Environment failures are excluded from this count.

C3 Feedback Behavior reports four quantities derived from the official score sequence $y_0,\dots,y_T$: peak retention $A_1$, dip rate $M/T$ where $M$ is the number of dip episodes, dip depth $\frac{1}{M}\sum_{e=1}^{M}(p_e - v_e)$ with $v_e$ the lowest score in episode $e$, and recovery credit $A_2$. These capture how well agents retain performance, how often they dip, how deep those dips are, and how much credit they earn for recovering.

Metric Definitions

Appendix details dip metrics, task‑level self‑improvement results, and the inter‑task experience protocol.

Dip depth captures the full peak‑to‑trough regression, while recovery credit uses the first dipped score to measure regained performance; both are computed only when a dip is observed.

The figure also reports the mean number T of evaluated commit rounds in gray, quantifying exposure without counting toward the C3 metric.

Additional diagnostics—peak position, monotonicity, trace coverage, final‑transition rise, and dip‑run count—remain in the detailed reproduction table but are omitted from the compact figure to avoid redundancy.

Task‑level outcomes reveal that model‑level means in Figures 7 and 8 hide substantial variation across tasks, prompting separate intra‑task (32 tasks) and inter‑task (19 targets) analyses using avg@3.

For intra‑task self‑improvement, every model shows more positive than negative outcomes; Kimi records 17 positive versus 10 negative despite a slightly negative mean.

In the inter‑task setting under avg@3, five models have more positives than negatives, whereas Gemini and LongCat exhibit the opposite pattern, aligning with their negative aggregate gains.

The rollout protocol reuses three baseline rollouts to obtain scores $S(0)$, extracts lessons from the best baseline trajectory into lessons.md, and then runs three augmented rollouts per target to obtain scores $S(+)$, with lessons transferred but not enforced.

Source tasks are chosen from four pilot models—Claude‑Opus‑4.7, GPT‑5.5, GLM‑5.2, and LongCat‑2.0—by requiring best@3_score > 0.5 and best@3_commits ≥ 5, then selecting the highest‑scoring task per category under avg(best@3_score × best@3_commits).

Target tasks comprise the remaining 32 candidates, filtered to exclude any with avg@3 ≥ 0.95 without lessons, leaving 19 targets such as `llm_online_serving`, `moving_mnist_world_model`, and `flash_attention`.

For each model–target pair we compute $M_{avg}=avg@3(S(+)) - avg@3(S(0))$ and $M_{best}=best@3(S(+)) - best@3(S(0))$, then average these metrics over the 19 retained targets.

Best@3 inter‑task results (Figure 16) show that GPT improves more on avg@3 (+0.063) than on best@3 (+0.022), while GLM (+0.040 vs +0.067) and Opus (+0.001 vs +0.038) gain more on best@3, highlighting distinct improvement profiles across metrics.

Experimental Procedures

Concrete task instructions, evaluation pipeline, and lessons from a trajectory‑derived run.

J.1 provides the full task instruction for the `grpo_multisource` Model Development task, including a placeholder @@ROOT@@ that marks the workspace root.

The setup table lists the training script, reward functions, entrypoint, and the base model (Qwen2.5‑VL‑7B, 4‑bit quantized), followed by three data sources: Geometry3K (~2400 samples), MathVision (~2000), and ChartQA (~1500), all in a unified “question, answer, image” format.

The goal is to maximize `mathvista_accuracy` on 100 held‑out MathVista problems, with a retention gate that zeroes the score if VQA accuracy drops more than 10 % relative to the base model.

Evaluation is invoked via python3 @@ROOT@@/app/`evaluate_local`.py; a quick‑check runs on 20 MathVista and 10 VQA items, where higher MathVista accuracy is better and the forgetting gate (VQA ≥ 0.9) nullifies the reward if violated.

Rules restrict edits to @@ROOT@@/app/train.py and @@ROOT@@/app/rewards.py, forbid changes to the original model or data directories, require a LoRA adapter saved under app/output/, prohibit external network access, limit execution to a single GPU, and cap the total wall‑clock time at 8 hours.

Long‑Horizon Automated Research and Development (R&D) Evaluation adds two requirements: fully exploit the allotted time budget before stopping, and if any environment‑level issue (missing files, hardware failure, etc.) blocks progress, report the command, error, and attempted fix immediately.

Version control must be initialized inside app/, with commits after each iteration describing the change and hypothesis; snapshots of each finished adapter are archived under @@ROOT@@/`output_snapshot`/_/, while a journal at app/output/journal.md records change, hypothesis, commands, results, and next decisions.

Before starting, the hardware (GPU, CPU, memory) is inspected and recorded in the journal, and the solution is adapted to the actual allocated resources.

J.2 presents lessons extracted from the highest‑scoring trajectory of a three‑rollout experiment on `data_select_ifeval`, showing a +0.26 increase in avg@3 and +0.66 increase in best@3 relative to the lesson‑free baseline.

The source context describes a data‑selection task where an agent must pick ≤ 5 000 samples from a 50 000‑sample pool (19 sources) to maximize IFEval prompt‑level strict accuracy after LoRA fine‑tuning on Qwen2.5‑3B‑Instruct, using a fixed recipe (LoRA rank 16, lr 1e‑4, 1 epoch, batch 8, cosine scheduler).

What Worked: (1) establishing a proper baseline before iterating; (2) switching from small‑sample to full evaluation; (3) using ultra‑minimal training as a diagnostic; (4) keeping a git‑based experiment journal; (5) systematically exploring selection axes.

Failures and Pitfalls: (1) trusting noisy evaluation for too long; (2) stale evaluation cache contaminating early results; (3) not recognizing the performance ceiling early enough; (4) over‑indexing on user‑prompt features without checking assistant‑response quality; (5) never attempting data augmentation.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers