SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring

Yuling Shi, Jinghan Xu, Kelin Fu, Wenhao Zeng, Shilin He, Lei Zhang, Yue Liu, Zelin Zhao, Terry Yue Zhuo, Jialun Cao, Siyu Ye, Tianyu Liu, Kai Cai, Shing-Chi Cheung, Xiaodong Gu

A multilingual, expert-curated benchmark for large-scale code refactoring that remains unsaturated by frontier AI agents.

How can we evaluate AI coding agents on complex, multi-file refactoring tasks that existing benchmarks fail to capture?

Existing coding benchmarks are rapidly saturating, and their evaluation quality is often compromised by flawed test suites and narrow, single-file tasks that fail to capture the complexity of real-world software engineering. The authors introduce SWE-Bench ProMax, a benchmark of 170 expert-curated, multi-file refactoring instances across seven programming languages, featuring rewritten specifications and manually verified test suites to ensure precise evaluation. Frontier models struggle with this task, with the best-performing model achieving only a 41.2% resolve rate, confirming that large-scale cross-file coordination remains a significant bottleneck for current AI agents.

Paper Primer

The benchmark addresses the "incomplete refactoring" failure mode: agents often identify the core logic changes but fail to propagate updates to peripheral call sites, documentation, and test fixtures. The core move is a rigorous, three-stage curation pipeline that filters out low-complexity tasks and rewrites issue descriptions to serve as unambiguous specifications for the required structural transformations.

SWE-Bench ProMax presents a meaningful, unsaturated challenge for frontier models.

The best-performing model (GPT-5.2) achieves a 41.2% resolve rate, significantly lower than the 75%+ rates seen on simpler benchmarks like SWE-bench Verified.

Open-weight models are highly cost-efficient alternatives to proprietary frontier models for refactoring tasks.

GLM-5 achieves a 36.5% resolve rate at \$0.24 per instance, compared to Claude Sonnet 4.6's 38.8% at \$4.77 per instance.

Why is refactoring a better test of agent capability than standard bug-fix benchmarks?

Refactoring requires coordinated, behavior-preserving changes across many files (averaging 11.4 files in this benchmark), forcing agents to maintain a coherent plan and manage cascading dependencies rather than just applying isolated patches.

How does this benchmark avoid the "flawed test" issues found in prior evaluations?

Every instance undergoes manual expert review to remove overly narrow tests that reject correct solutions and overly broad tests that check unstated requirements, ensuring the test suite is a necessary and sufficient condition for the gold patch.

Researchers should shift focus from isolated code generation to multi-file coordination, as the primary bottleneck for agents is no longer reasoning ability, but the capacity to persist through large-scale structural changes without entering unproductive edit-revert cycles.

The Need for Complex Benchmarking

We expose benchmark saturation and introduce a high‑complexity refactoring benchmark.

AI coding agents have rapidly improved, but current benchmarks focus on isolated, small‑scale tasks, leading to saturation and questionable evaluation quality.

We need a benchmark that forces agents to handle large, multi‑file refactoring tasks, exposing genuine software‑engineering reasoning beyond narrow test suites.

**Figure 1** Distribution of modified files (left) and lines of code (right) per instance across benchmarks. SWE-Bench ProMax instances are substantially larger: 30% modify more than 10 files and 32% change over 200 lines of code, while 86% of SWE-bench Verified instances modify only a single file.

Benchmark saturation and evaluation quality drive the need for a high‑complexity refactoring benchmark.

Positioning Against Existing Benchmarks

We situate SWE‑Bench ProMax among prior code benchmarks and refactoring datasets.

Recent code‑generation benchmarks have saturated on isolated, single‑file tasks, prompting the community to seek richer, multi‑file challenges.

SWE‑bench Verified is a repository‑level benchmark of refactoring tasks where each instance has been automatically verified against a gold patch.

Constructing SWE-Bench ProMax

This section details the SWE‑Bench ProMax benchmark, its task definition, construction pipeline, and dataset composition.

SWE‑Bench ProMax is a benchmark of multi‑file refactoring tasks that require agents to modify dozens of files in a realistic codebase.

How does SWE‑Bench ProMax differ from earlier SWE‑bench variants?

Earlier variants target single‑file edits or small functions, while ProMax requires coordinated changes across many files, a realistic test of software‑engineering reasoning.

A Refactoring Instance bundles a repository snapshot, a natural‑language issue description, a test suite, and a gold patch that together define a concrete refactoring problem.

Why is the gold patch included instead of just the final code?

The gold patch serves as a ground‑truth reference that can be compared against the agent’s output and used to verify that the test suite indeed captures the intended refactoring.

Stage 1 – Data collection: query the GitHub API for repositories with ≥500 stars, an approved license, and a primary language covering ≥80 % of the codebase.

Extract commits after January 2025 whose messages contain “refactor” but not “bug fix”, and that modify both test and non‑test files.

Stage 2 – Environment construction: for each candidate commit, build an isolated Docker container at the pre‑refactoring commit, install all build dependencies, apply the gold patch, and run the full test suite.

Discard any candidate where the environment cannot be built or the gold patch fails the tests.

Stage 3 – Filtering and problem rewriting: (1) Experts, aided by LLMs, analyze the commit diff to understand scope and intent.

(2) Quality filtering removes single‑file or trivially small changes and eliminates overly narrow or overly broad test suites.

(3) Problem statements are rewritten from scratch with LLM assistance to produce precise, self‑contained specifications.

(4) A final expert verification ensures consistency among description, test suite, and gold patch.

Stage 1 collects the repository because it has 600 stars and the primary language is Python.

Stage 2 builds a Docker container, installs dependencies, applies the gold patch, and the test suite (a single assert checking

Stage 3’s experts rewrite the issue description to: “Add a helper function to utils.py and update main.py to use it, preserving existing behavior.”

Human verification confirms that the rewritten description, test suite, and gold patch are mutually consistent.

This concrete trial shows how a multi‑file change is turned into a self‑contained benchmark instance, illustrating the interaction between data collection, environment validation, and expert curation.

**Figure 2.** A representative SWE-Bench ProMax instance: refactoring NASA’s F’Prime flight software framework requires coordinated changes across 244 files spanning the entire codebase.

**Figure 3.** Data collection and curation pipeline for SWE-Bench ProMax.

**Table 2.** Dataset statistics of SWE-Bench ProMax

Performance of Frontier Agents

Key performance numbers, language breakdowns, and cost trade‑offs for SWE‑Bench ProMax.

GPT‑5.2 attains only 41.2% resolve rate on SWE‑Bench ProMax, far below the >75% achieved on SWE‑bench Verified.

Table 3 reports an overall resolve rate of 41.2% for GPT‑5.2.

All evaluated agents find the benchmark challenging, confirming that multi‑file refactoring remains an unsolved problem.

Resolve Rate is the percentage of benchmark instances where the agent produces a correct, behavior‑preserving refactoring patch.

TypeScript and Rust exhibit the widest variance: Claude Sonnet 4.6 reaches 53.6% on TypeScript and 63.6% on Rust, while Gemini‑3‑Pro scores 0.0% on TypeScript and Kimi‑K2.5 only 18.2% on Rust, suggesting language‑specific data rather than intrinsic difficulty.

**Figure 5** Left: Cumulative distribution of files modified by agents (Claude Sonnet 4.6, Kimi-K2.5) versus the gold patch. Agents modify fewer files than required overall, with the gap widening for larger patches. Right: Cumulative distribution of interaction rounds on resolved (solid) versus unresolved (dashed) instances. Failed attempts consume substantially more rounds than successful ones.

These patterns point to a dominant failure mode: agents often locate the core files to edit but stop short of propagating the change to all dependent call sites, docs, configs, and tests, causing the test suite to fail despite correct core logic.

Higher API cost does not translate to proportionally higher resolve rates: Claude Sonnet 4.6 costs \$4.77 per instance yet trails GPT‑5.2, whereas GLM‑5 achieves 36.5% at a twentieth of that cost.

Task Category and Skill Analysis

Task categories reveal how SWE‑Bench ProMax blends refactoring, API changes, and bug fixes into multi‑file challenges.

The benchmark’s task taxonomy shows that most instances combine several software‑engineering concerns, not just isolated refactoring.

**Figure 6** Distribution of task categories across SWE-Bench ProMax instances (multi-label; percentages sum to more than 100%). The dominant categories—Refactoring Cleanup (66.5%) and API Interface Change (65.3%)—confirm the refactoring focus of the benchmark, while the substantial presence of New Feature (43.5%) and Bug Fix (41.2%) reflects the multi-faceted nature of real-world code restructuring.

**Figure 7.** Number of categories per instance. Every instance involves at least two categories, and nearly half (46.5%) involve three or more simultaneously, indicating that SWE-Bench ProMax tasks require holistic software engineering skills rather than isolated refactoring ability.

**Figure 8** Co-occurrence matrix of task categories. API Interface Change and Refactoring Cleanup co-occur in 79 instances, while Bug Fix frequently accompanies both (33 and 48 instances respectively), reflecting how refactoring often surfaces latent defects that must be addressed concurrently.

**Figure 9** Distribution of required reasoning skills across SWE-Bench ProMax instances (multi-label). Nearly all instances require cross-file reasoning (99.4%) and API semantics understanding (98.8%), confirming that the benchmark systematically tests sustained multi-file comprehension. Pattern matching (91.8%) and interface contract reasoning (97.1%) are also near-universal, while data flow (88.8%), domain knowledge (79.4%) and type system reasoning (50.6%) appear in the majority of instances.

Dataset Composition and Statistics

Appendix A details the composition and statistics of the SWE‑Bench ProMax dataset.

The dataset comprises 170 high‑complexity refactoring instances sourced from 70 open‑source repositories spanning seven languages.

The three‑stage pipeline first filters by commit size and language, then applies static analysis to ensure compilability, and finally selects candidates that exhibit non‑trivial cross‑file changes.

Expert curators rewrite issue descriptions to remove ambiguity and manually review test suites to avoid overly narrow or overly broad checks.

**Figure 4.** Language distribution in SWE-Bench ProMax.

**Table 4.** Repositories and licenses used in SWE-Bench ProMax, grouped by language. 70 repositories across 7 languages.

**Table 5** Per-language statistics for SWE-Bench ProMax, including number of repositories, instances, and average patch complexity.

Representative Instance Examples

Representative SWE‑Bench ProMax instances showcase diverse refactoring scales across languages.

This appendix lists a single representative refactoring instance for each language in SWE‑Bench ProMax, highlighting repository size, modified files, and the core problem each instance addresses.

The monolithic

Any edit to a single alias or configuration triggers recompilation of the entire framework.

Resulting build cycles become significantly longer during iterative development.

Conflating multiple concerns in one header creates workflow friction and hampers developer productivity.

The Gantt diagram engine operates at day‑level resolution, treating each calendar day as indivisible.

Tasks requiring sub‑day precision (e.g., 9 AM–3 PM) cannot be represented directly.

Workarounds split a single day into artificial multiple days, causing duration and resource‑load inaccuracies.

This coarse temporal granularity limits precise scheduling and integration with tools that use finer time units.

The

These names suggest PWM‑only scope, yet they control settings for all motor protocols, including digital ones like Dshot.

Developers misinterpret the fields, leading to configuration errors for non‑PWM protocols.

Misleading field names create cognitive friction and bugs when configuring digital motor protocols.

Scalar arguments to tensor ops (e.g.,

Autodiff checkpointing must store and replay these scalars without a unified runtime representation.

Each new scalar‑accepting operation requires duplicated conversion code across multiple backends.

Backend‑specific scalar types cause pervasive type‑mismatch friction and boilerplate across the codebase.

File uploads span multiple storage drivers (115 Cloud, 123 Cloud, Google Drive, OneDrive, etc.).

Stream handling, caching, and buffer management differ across drivers, leading to reliability issues.

These issues propagate through several interrelated subsystems, degrading overall upload stability.

Inconsistent stream and buffer handling across drivers creates critical reliability problems.

The library supports multiple language model providers, each with distinct output formats (JSON, YAML) and wrapper conventions.

Logic for format decisions is scattered across prompt generation, model configuration, resolver parsing, and provider‑specific code.

This fragmentation leads to subtle integration and maintenance challenges.

Distributed format‑handling logic hampers integration and increases maintenance burden.

Components such as Modal, Drawer, Collapse, Tabs, Tooltip, Dropdown, and Image provide an “unmount when hidden” feature.

The API for this feature varies inconsistently across components.

Inconsistent APIs hinder developers from reliably optimizing performance and managing component state.

API inconsistency across components impedes performance optimization and clean state management.

**Table 6.** Summary of representative instances in SWE-Bench ProMax, one per language, sorted by number of modified files.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers