SWE-Bench Science: Can Coding Agents Resolve Engineering Tasks in Science?

Zhipeng Xu, Jiahao Lu, Yining Zheng, Yuxin Wang, Xipeng Qiu

SWE-bench Science evaluates coding agents on 119 real-world scientific software engineering tasks across 20 domains.

Can current coding agents reliably resolve complex engineering tasks within scientific software, and how does scientific domain knowledge influence their performance?

Scientific software is now a core instrument for research, yet coding agents often fail to maintain the scientific validity of the code they repair. The authors introduce SWE-bench Science, a repository-level benchmark of 119 tasks across 20 scientific domains, categorized into issue-driven, expert-exploratory, and engineering-integration paradigms. Even the best-performing agent achieves a Pass@1 below 50%, revealing that current models struggle with scientific abstraction, system-wide integration, and generalizing principles beyond observed cases.

Paper Primer

The benchmark uses a "Chain-of-Evidence" protocol to isolate scientific contracts from engineering context. It forces agents to navigate production repositories, modify interacting modules, and satisfy hidden scientific validators that test boundary conditions and semantic equivalence rather than just public test cases.

Scientific auxiliary information does not uniformly improve performance.

A paired ablation shows that while extra guidance can improve token efficiency and average scores, it can also induce "anchoring" and decrease Pass@1 for stronger models. For GPT-5.6-sol, Pass@1 dropped from 36.26% to 31.87% when provided with explicit scientific rationale.

Coding agents exhibit four recurring failure mechanisms in scientific contexts.

Manual audit of unsuccessful repairs identifies deficits in scientific abstraction, misguided surface-level fixes, incomplete system integration, and failures to generalize scientific principles. Claude-Opus-5, the top performer, still recorded 58 categorized scientific errors across the 119 tasks.

Why is this benchmark necessary if general-purpose coding benchmarks like SWE-bench already exist?

General benchmarks focus on aggregate task success, which masks why agents fail in science. Scientific software requires preserving specific physical or mathematical invariants across complex, multi-file systems, which standard unit tests often fail to capture.

What distinguishes the three task paradigms in this benchmark?

Issue-driven tasks focus on repairing known bugs; Expert-exploratory tasks require autonomous reasoning about unknown root causes; and Engineering-integration tasks test the ability to maintain system-wide architecture and data-flow consistency.

Scientific software engineering requires more than just code generation; it demands that agents perform disciplined exploration and validate repairs against scientific invariants. Future improvements must focus on better scientific abstraction and autonomous verification rather than just increasing model scale.

Introduction and Motivation

We expose why coding agents struggle with scientific software and introduce a benchmark to study these failures.

Software is now an integral part of the scientific instrument; a defect can corrupt both program output and the underlying evidence.

Current evaluations of coding agents emphasize aggregate task success, which reveals little about why agents fail on scientific software.

Reliability means the software consistently yields correct scientific results, so any bug can undermine the evidence supporting a claim.

To study these challenges we introduce SWE‑bench Science, a repository‑level benchmark covering scientific software engineering.

The suite comprises 119 tasks drawn from 98 GitHub repositories spanning 20 scientific domains, organized into Issue‑driven, Expert‑exploratory, and Engineering‑integration paradigms.

Our analysis uncovers four recurring failure mechanisms: missing scientific knowledge or abstraction, misguided surface‑level exploration, incomplete repair coverage across system components, and inability to generalize scientific principles beyond observed cases.

We conduct a paired ablation that removes explicit scientific guidance while preserving the repository and executable engineering context, revealing that scientific knowledge is not uniformly beneficial.

Well‑grounded scientific information can constrain repair, improving average performance and token efficiency, whereas poorly aligned guidance may induce anchoring and does not necessarily boost exact repair success.

**Table.** Summary of scientific software tasks, including their domain, repository, issue/PR references, knowledge scope, and specific task focus.

The critical role of code in scientific evidence demands reliable engineering practices.

Benchmark Overview

We present SWE‑bench Science and situate it among existing software‑engineering and scientific‑workflow benchmarks.

We introduce SWE‑bench Science, a repository‑level benchmark that evaluates coding agents on real scientific software tasks across many domains.

A large‑scale benchmark of 119 tasks drawn from 98 distinct GitHub repositories, covering 20 scientific domains, designed to test agents on three complementary paradigms of scientific software maintenance.

Benchmark Statistics

Benchmark statistics reveal expanded task coverage and domain breadth.

SWE‑bench Science expands benchmark coverage to 119 tasks, surpassing prior suites.

Compared to SciCode’s 80 tasks and MMSciCode’s 6 tasks, SWE‑bench adds 39 more tasks.

The input size metrics (non‑empty lines, Patch +, Patch –) remain comparable across benchmarks.

**Figure 1.** Distribution of scientific domains and task paradigms, along with statistics for input non-empty lines, Patch +, and Patch -.

Task Schema and Evaluation

We detail the benchmark composition, task schema, and evaluation pipeline.

This section enumerates the benchmark’s scale, the distribution of task types, and the exact fields presented to the coding agent versus the evaluator.

Sample a repository snapshot, lock its dependencies, and strip history to create a reproducible workspace.

Attach a frozen problem statement that describes the software defect without revealing the reference patch.

Provide the required scientific context $c_{\text{req}}^i$, a fixed set of domain definitions and constraints for the task.

Expose public tests that encode software and scientific assertions usable for interactive debugging.

The coding agent iteratively generates a patch against the public tests within the sandbox.

Upon submission, the evaluator mounts private tests and other hidden fields in a separate container to verify correctness.

Metrics recorded include input code size, patch addition/deletion line counts, and success measured by Pass@1.

The agent receives the snapshot and problem statement, then reads the required scientific context defining the conversion factor.

It generates a patch adding 12 lines and deleting 3 lines to implement the conversion.

Public tests pass; the evaluator then runs private tests, confirming the patch’s correctness.

This concrete run shows how patch line counts (addition = 12, deletion = 3) relate to the benchmark’s reported averages.

**Figure 3.** Evaluation pipeline in SWE-bench Science for an example task. The diagram shows the frozen agent-visible inputs, the agent-driven coding loop, and the clean evaluation phase with separate public diagnostics and private scientific cases.

Overall, the benchmark varies repository scale and patch size while keeping the scientific context and public tests constant, enabling measurement of agent performance via Pass@1.

Task Construction

Describes the four‑stage pipeline for building raw tasks and their redesign into three benchmark paradigms.

The evaluation pipeline proceeds in three phases: the agent receives the task inputs, runs a coding loop to propose a fix, and finally undergoes a clean evaluation against public and private checks.

Stage 1 Source Sampling and Screening – collect real issues, PRs, and commits from scientific‑computing repos and discard tasks that are trivial, unstable, or overly similar to existing samples.

Stage 2 Snapshot Freezing and Reproduction – freeze the code snapshot at the point of the defect ($\mathcal{S}_{\text{bug}}$) and verify in an isolated container that the anomaly reproduces reliably.

Stage 3 Public‑Material Abstraction and Information Isolation – extract the scientific invariants, build a public evaluation package (repo, description, reproduction script) that omits the patch location and hidden assertions.

Stage 4 Hidden Oracle and Counter‑Calibration – generate hidden validation suites that test semantic equivalence, boundary conditions, and resistance to heuristic or incomplete fixes.

Running the public script yields $E_k = -75.1234$ Hartree for the primitive‑cell k‑point mesh.

The hidden oracle computes the reference energy $E_Γ = -300.4936$ Hartree for the $\Gamma$ supercell; with $N=4$ primitive cells, the expected per‑cell energy is $E_Γ/N = -75.1234$ Hartree.

The public test checks $ΔE = |E_k - E_Γ/N| = 0$, which satisfies the $5\times10^{-5}$ Hartree threshold.

One hidden test perturbs the k‑point order, causing $E_k$ to shift to $-75.1240$ Hartree; $ΔE$ becomes $6\times10^{-5}$ Hartree, failing the hidden oracle.

This concrete run shows how the public evaluation can be passed while a subtle hidden change reveals an incomplete repair, illustrating the need for both public and private checks.

The benchmark then classifies raw tasks into three paradigms: Issue‑driven tasks focus on localized bug fixes, Expert‑exploratory tasks require autonomous investigation of scientific discrepancies, and Engineering‑integration tasks assess end‑to‑end workflow completion.

**Figure 4.** Combined overview of raw-task construction and class-specific task re-design.

Task Paradigms

Experimental setup isolates distinct agent capabilities via two task families.

The experimental setup distinguishes three task families to isolate distinct agent capabilities. Issue-driven tasks target known bugs, while expert-exploratory tasks require reasoning about unknown causes. Engineering-integration tasks are described later.

These tasks ask an agent to repair a concrete, historically observed defect while keeping the surrounding code unchanged.

How do Issue-driven tasks differ from standard bug‑fixing benchmarks?

Standard benchmarks often expose the full repository and expect the agent to locate the bug among many files. Issue-driven tasks isolate a single historical defect, provide a minimal reproducible example, and use hidden validators to test whether the fix restores the scientific semantics rather than merely passing the public script.

These tasks require the agent to discover the underlying scientific cause of an anomaly without being given the exact faulty code.

Why aren’t Expert-exploratory tasks just harder versions of Issue-driven tasks?

Issue-driven tasks start from a known defect and expose a minimal example, whereas Expert-exploratory tasks begin with a broad scientific scenario, hide the exact faulty code, and require the agent to hypothesize and test root causes. The validation also varies physical or computational parameters, demanding genuine scientific reasoning rather than simple bug‑fixing.

Engineering Integration and Context

This section details the engineering‑integration task definition and experimental setup for measuring scientific auxiliary information impact.

These tasks require an agent to understand and modify the whole system architecture, not just fix isolated bugs.

Auxiliary information supplies domain‑specific scientific rationale that is not inferable from the code or its runtime behavior.

Pipeline Analysis and Gap Selection – examine the repository’s call chain and pick a functional gap spanning multiple modules.

Preservation of Engineering Context – keep the full package structure and neighboring modules in the public source, exposing the agent to realistic navigation challenges.

End‑to‑End Contract Validation – use hidden validators (alternative paths, state‑reset tests, inter‑module contracts) to verify that the repair achieves system‑level integration.

Step 1 selects the gap: the repository lacks a

Step 2 preserves context: both modules remain unchanged, and the agent sees the full directory layout (

Step 3 validates the repair: a hidden validator runs

This concrete trial shows how the agent must insert a new module‑level check, navigate across files, and satisfy a hidden validator that tests a property not observable from the public tests alone.

Experimental Results

Experimental comparison of models on SWE‑bench Science across key metrics.

We evaluate eight coding agents on the SWE‑bench Science benchmark, reporting several metrics that capture public, private, and task‑specific performance.

Claude‑Opus‑5 achieves the highest overall Pass@1 on SWE‑bench Science.

Table 2 shows Claude‑Opus‑5 at 47.90% overall Pass@1, the next best model (GPT‑5.6‑sol) at 46.22%.

**Figure 1** Pass@1 comparison of coding agents on SWE-bench Science.

DeepSeek‑V4‑Pro attains a perfect public score, while GPT‑5.6‑sol dominates the private, Fail2Pass, and Pass2Pass metrics.

**Figure 5.** Pass@1 versus mean token consumption per task over the common 119-task evaluation. Dashed guides connect each observation to the corresponding axes.

Failure Analysis

6 Analysis

6.1 Observed Failure Mechanisms

Table 3 Error‑count breakdown by coding agents over the common 119 tasks. The total column records attempts assigned to the four mutually exclusive scientific failure mechanisms defined in the text; for Claude‑Opus‑5, the parenthesized +4 denotes additional runtime or evaluation‑path failures. The four category counts sum to the main total for each row. The lowest count in each column is bold and underlined.

| LLM | Harness | Total errors | Knowledge/abstraction | Exploration/surface repair | Repair coverage/system integration | Scientific generalization | |----------------------------------|---------|--------------|-----------------------|----------------------------|-------------------------------------|----------------------------| | GPT‑5.6‑sol (max) | – | 64 | 69 | 18 | 10 | 22 | | Claude‑Opus‑5 (max) (+4) | – | 77 | 24 | 2 | 2 | 21 | | DeepSeek‑V4‑Pro (max) | – | 81 | 15 | 12 | 14 | 19 | | Kimi‑K3 (max) | – | 90 | 20 | 14 | 14 | 22 | | GLM‑5.2 (max) | – | 91 | 20 | 14 | 14 | 24 | | Nex N2 | – | 102 | 31 | 15 | 15 | 23 | | DeepSeek‑V4‑flash (max) | – | – | – | – | – | – | | Claude Code | – | – | – | – | – | – | | Qwen3.5‑397B | – | – | – | – | – | – | | Codex | – | – | – | – | – | – | | Claude Code 58 (+4) | – | – | – | – | – | – | | Claude Code | – | – | – | – | – | – | | Kimi Code | – | – | – | – | – | – | | Codex | – | – | – | – | – | – | | Codex | – | – | – | – | – | – |

We use four recurring scientific failure mechanisms in the audit reports.

* **Scientific‑knowledge or abstraction deficit** denotes a repair based on an incorrect or incomplete scientific object, mathematical definition, or domain abstraction. * **Misguided exploration or surface‑level repair** denotes a patch that addresses the visible symptom or public metric without tracing the failure to an independent oracle or the underlying scientific contract. * **Incomplete repair coverage or system integration** denotes a locally plausible repair that does not satisfy the requirements of the full software system; for example, one module is corrected while its interactions, data flow, shared invariants, or compatibility with other modules remain unpreserved. * **Failure of scientific‑knowledge generalization** denotes a repair that handles the observed scientific case but does not extend the same scientific principle to unseen conditions, equivalent representations, boundary regimes, or other variants that require scientific generalization.

Claude‑Opus‑5 additionally has four runtime or evaluation‑path failures, in which an attempt does not complete the intended execution or evaluation path and therefore cannot be attributed to one of the four scientific mechanisms.

Claude‑Opus‑5 produces the lowest categorized scientific‑error count (58, plus 4 runtime or evaluation‑path failures) and the fewest misguided‑exploration or surface‑level‑repair errors (2). DeepSeek‑V4‑flash (max) records the fewest scientific‑knowledge generalization errors (6), while DeepSeek‑V4‑Pro records both the fewest scientific‑knowledge or abstraction errors (15) and the fewest incomplete‑repair or system‑integration errors (19).

Impact of Scientific Information

We report how scientific auxiliary information changes model scores, Pass@1, and token usage.

Scientific auxiliary information improves DeepSeek‑V4‑flash Pass@1 but harms GPT‑5.6‑sol Pass@1, revealing model‑dependent effects.

DeepSeek‑V4‑flash Pass@1 rises from 16.48 % to 23.08 % (+6.6 %) while GPT‑5.6‑sol Pass@1 falls from 36.26 % to 31.87 % (‑4.4 %).

Scientific information can provide semantic constraints such as limiting cases and coordinate consistency, but it may also introduce anchoring or scope spillover, leading to premature reliance on supplied explanations. Consequently, weaker models like DeepSeek‑V4‑flash gain more from the auxiliary data, whereas stronger models like GPT‑5.6‑sol see diminished Pass@1.

**Figure 6** Scores and token consumption over the 91 tasks whose scientific-information content differs between conditions. GPT-5.6-sol and DeepSeek-V4-flash are each shown with and without scientific information on the same task subset.

**Figure 7.** Task-level overlap of Pass@1 success on the 91-task separable subset. (a) GPT-5.6-sol passes 29 tasks with scientific information and 33 tasks without it; 21 pass in both conditions, 8 only with information, and 12 only without it. (b) DeepSeek-V4-flash passes 21 tasks with scientific information and 15 tasks without it; 12 pass in both conditions, 9 only with information, and 3 only without it.

Across eight coding‑agent configurations, the best model achieves 47.90 % Pass@1 while scoring 96.64 % on public tests.

These results highlight a large gap between visible‑test performance and full correctness.

Domain Coverage

Domain coverage of the task inventory.

The task inventory spans a broad set of scientific domains, with each domain contributing a distinct number of tasks. Below we list the domains and their task counts.

Task Inventory Part 1

This section details the full task inventory, linking each task to its scientific domain, repository, and knowledge scope.

The benchmark inventory enumerates 119 tasks (IDs 001–119), each assigned to one of 20 scientific domains and linked to its upstream repository, issue or pull request, knowledge scope, and task focus.

The table lists 17 tasks (004-020) across various scientific domains, including Biology, Biomedical Engineering, Physics, Astronomy, and Materials Science and Engineering. It details the upstream repository, associated Issue/PR numbers, the specific knowledge scope of the task, and the technical task focus for each entry.

Three recurring annotation pitfalls are highlighted: (1) rotational‑symmetry must be inferred from the true transition‑state graph, not from atom ordering; (2) range‑separated tuning requires consistent unit conversion for density cutoffs; and (3) extxyz parsing demands proper stress‑to‑virial conversion to preserve tensor semantics.

Task Inventory Part 2

A catalog of scientific software tasks with domains, repositories, and specific issues.

The following tables enumerate the scientific software tasks considered in this study.

Task Inventory Part 3

Catalog of scientific software tasks, their repositories, and detailed issue specifications.

**Table 5.** Task-level scientific-domain, provenance, and scientific-knowledge inventory. A missing issue or pull-request reference indicates that no reference was provided in the source annotation.

**Table.** Summary of scientific software tasks, repositories, and associated technical issues.

**Table.** Summary of scientific software tasks, repositories, issues/PRs, knowledge scope, and task focus.

Task Inventory Part 4

Catalogues the diverse scientific‑software tasks, their domains, repositories, and concrete technical constraints.

This inventory enumerates the concrete software‑engineering challenges that arise when scientific code is integrated into real‑world repositories.

The provided image is a table listing various scientific software tasks, their associated domains, repositories, issue/PR identifiers, knowledge scopes, and specific task focuses.

**Table.** Summary of scientific software tasks, repositories, issues/PRs, knowledge scope, and task focus.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers