ASI-Bench: At the Dawn of Artificial Superintelligence

Junwei Zhou, Zhen Sun, Binyu Li, Jiangyu Zhou, Yuexi Pan, Hengyu Wang, Honghe Ren, Xiaohan Jia, Xueyang Zhou, Xiaoyu Cao, Yongchao Chen, Yuanning Feng, Junhao Wu, Cheng Zhang, Sijia Chen, Haoyu Xue, Chengsong You, Huan Wang, Koutian Wu, Peigan Gao, Jiakun Wu, Wenzhe Li, Ergan Shang, Qingyuan Zheng, Jingjing Zhou, Ruixuan Jia, Yan Xu, Hongrui Zhang, Xiao-Han Ma, Zhengxiang Cheng, Yuexing Hao, Liting Mai, Xianglin Ji, Wenjun Zhang, Zhuofan Chen, Yixiao Huang, Chi Wang, Wenyue Hua, Yilun Hao, Yuantao Zhai, Ziyan Zhao, Jingyan Xie

ASI-Bench evaluates AI systems on project-level scientific research as human methodological guidance is progressively withdrawn.

How do current AI agents perform on open-ended, project-level scientific research tasks that require autonomous knowledge creation rather than just information retrieval?

Current AI benchmarks primarily test knowledge recall or the execution of human-specified procedures, leaving it unclear whether models can independently conduct open-ended scientific research. ASI-Bench introduces a "guidance gradient" across 60 project-level tasks, where the same research problem is presented with varying levels of human instruction—from full procedural guidance to minimal objective-only prompts. Performance drops sharply as guidance is removed, revealing that current systems are highly dependent on human-designed workflows rather than capable of autonomous scientific discovery.

Paper Primer

The benchmark uses a four-level (B1–B4) information gradient to isolate where AI research capabilities break down. B1 provides the full method and solver steps; B2 provides the method class but leaves implementation to the agent; B3 provides only the objective and data; and B4 adds task-irrelevant information to test robustness against distraction.

Current AI systems are significantly more dependent on procedural guidance than on method selection.

Average performance across 18 configurations drops by 21.82 points when procedural steps are removed (B1 to B2), but only by an additional 2.48 points when the method itself is removed (B2 to B3). The average score falls from 50.91 (full guidance) to 26.62 (no methodological guidance).

Increased computational budget does not guarantee stronger scientific autonomy.

Configurations with similar performance show massive cost disparities, and higher spending on inference-time reasoning (e.g., GPT-5.6 Sol Ultra) yields only moderate gains in autonomous research tasks. The best-performing configuration achieves a B3 score of 51.60, while the average across all configurations is 26.62.

Why does the benchmark include a B4 condition with irrelevant information?

B4 tests whether an agent can maintain its research direction when presented with plausible but distracting context, ensuring that performance drops in B3 are due to a lack of autonomy rather than simple sensitivity to noise.

What does this benchmark measure that existing scientific benchmarks miss?

Most existing benchmarks evaluate isolated components like coding or hypothesis generation; ASI-Bench evaluates the ability to sustain an end-to-end, multi-stage research project while independently determining the path to a solution.

Progress toward artificial superintelligence requires moving beyond "procedure following" toward autonomous workflow construction; researchers should prioritize improving the agentic harness and reasoning systems over simple scaling.

The Frontier of Scientific Autonomy

Introducing ASI‑Bench to expose the gap between knowledge retrieval and autonomous scientific creation.

Artificial Superintelligence (ASI) agents today excel at retrieving existing knowledge but still rely heavily on human‑provided methodological guidance. To assess whether they can move toward genuine scientific autonomy, we need a benchmark that forces agents to explore, invent, and verify new ideas without hand‑holding.

ASI‑Bench is designed to expose how far current AI systems can progress from merely recalling known facts to autonomously inventing and validating new scientific knowledge.

Across 18 state‑of‑the‑art agent‑model configurations, the average score falls from 50.91 with full methodological guidance to 29.10 when only the method is specified, and further to 26.62 when agents must determine the method themselves. This sharp decline quantifies the heavy dependence on human guidance and highlights the substantial gap toward true autonomous scientific research.

The key shift to watch is moving from knowledge retrieval to knowledge creation.

Designing the ASI-Bench Framework

Design of ASI‑Bench details tasks, guidance levels, and evaluation metrics.

Benchmark Design specifies the tasks, guidance conditions, and metrics that together assess scientific autonomy.

Four graded conditions control how much human methodological information is supplied to the agent, from full method description (B1) down to only the research goal and data (B3), with B4 adding irrelevant distractors.

How do the B1–B4 levels differ from typical “prompt engineering” tricks?

Prompt engineering usually tweaks wording to coax a model toward a desired answer, but it still leaves the underlying method fixed. B1–B4 explicitly remove or alter the method itself, forcing the agent to generate or select its own algorithm rather than follow a prescribed one.

Collect 60 project‑level research tasks spanning 11 scientific domains.

For each task, fix the research objective, input data, required outputs, and scoring rubric.

Run the task under four guidance conditions (B1‑B4) while keeping everything else identical.

Record interaction turns, execution steps, and total wall‑clock time.

Compute a performance score for each guidance level and aggregate across tasks.

The agent ingests the raw trajectory data and infers the underlying PDE structure.

It selects a finite‑difference scheme, implements the solver, and runs a simulation.

After the first simulation fails to match the data, the agent diagnoses the error, refines the discretization, and reruns.

Finally, it generates predictions for the held‑out time window and submits them for scoring.

Even without any methodological hint, the agent must discover the model, choose a numerical method, and iteratively improve its solution.

**Figure 1.** Overview of ASI-Bench. Left: B3 performance across agents. Right: scores from B1 to B4, where B1 provides full methods, B2 only the method name, B3 only the research goal and data, and B4 further adds distractors.

**Figure 2.** Performance comparison across HLE, SWE-bench, Terminal-Bench, and ASI-Bench.

Performance on Scientific Tasks

We report limited scientific autonomy and strong reliance on detailed methodological guidance across 18 agents.

Current systems show limited scientific autonomy without explicit methodological guidance.

Only Codex with GPT‑5.6 Sol (ultra) exceeds 50 on B3, achieving a score of 51.60.

Scientific autonomy is the ability of an AI system to independently formulate, select, and execute research methods without human‑provided step‑by‑step guidance.

**Figure 3.** Representative project-level tasks in ASI-Bench across physics, astronomy, electrical engineering, and computer science.

Current systems show limited autonomy without explicit methodological guidance.

Agent and Harness Performance

Evaluating how harnesses alter model capability on scientific tasks.

Using different harnesses can shift a model’s scientific capability by several points.

MiMo V2.5 Pro rises from 16.17 (MiMo Code) to 23.25 (Claude Code), and Kimi K2.7 from 19.72 to 27.34.

The remaining models show smaller or negligible changes, e.g., Kimi K3 moves only from 36.22 to 37.09, indicating that the harness effect is not uniform across architectures.

Case Study: Anisotropic Stiff Dynamics

Analyzes four prompt variants to show how guidance level shapes the agent’s scientific workflow.

The central premise is that agents must move from pure data retrieval toward autonomous scientific discovery. This case study compares four prompts that vary the amount of methodological guidance provided.

This prompt supplies the exact governing PDE together with a complete Fourier‑spectral solver recipe, letting the agent reconstruct the entire simulation pipeline.

How does the ETDRK4 scheme differ from a standard explicit Runge‑Kutta method for this PDE?

ETDRK4 integrates the linear part exactly via matrix exponentials ($E=\\exp(\\hat{L}\\,dt)$) and only approximates the nonlinear term, whereas an explicit Runge‑Kutta would treat the entire right‑hand side explicitly, forcing a tiny $dt$ because of the fourth‑order stiffness.

Compute wavenumbers: $k_x = 2\\pi\\,\\text{fftfreq}(4,dx=\\frac{2\\pi}{4}) = [-\\pi, -\\frac{\\pi}{2}, 0, \\frac{\\pi}{2}]$, similarly for $k_y$.

Form the linear operator $\\hat{L}= -\\alpha + k_x^2 + \\mu k_y^2 - \\nu k_x^4 - 2\\gamma k_x^2 k_y^2 - \\delta k_y^4$ for each $(k_x,k_y)$ pair.

Given a field $u$, compute its gradients $u_x, u_y$ via inverse FFT of $i k_x \\hat{u}$ and $i k_y \\hat{u}$.

Evaluate the conserved nonlinear term $f = (\\lambda_{xx}/2)u_x^2 + \\lambda_{xy}u_x u_y + (\\lambda_{yy}/2)u_y^2$, then compute $\\hat{N}=-(k_x^2+k_y^2)\\,\\text{fft2}(f)$ and apply the 2/3‑rule mask.

Pre‑compute $E=\\exp(\\hat{L}\\,dt)$, $E2=\\exp(\\hat{L}\\,dt/2)$, and the Kassam–Trefethen coefficients $Q,f_1,f_2,f_3$ (using contour integrals).

Perform one ETDRK4 step: update $\\hat{u}$ to $\\hat{u}_{\\text{next}}$ using the formula in the prompt.

Even on a $4\\times4$ grid the spectral pipeline reproduces the exact linear dynamics, while the nonlinear term is captured without aliasing thanks to the 2/3‑rule.

This prompt omits the explicit equation but describes the physical ingredients and suggests families of numerical solvers, requiring the agent to choose a compatible method.

Why is a Fourier pseudospectral discretization preferred over a finite‑difference stencil for this PDE?

On a periodic domain the Fourier basis yields exact derivatives for any mode, so the fourth‑order terms are represented without truncation error. Finite‑difference stencils approximate derivatives locally and would need a very fine grid to achieve comparable accuracy, especially for high‑order derivatives.

Spectral: compute $\\hat{u}$ via FFT, multiply by $i k_x$ (or $i k_y$) to obtain $\\widehat{u_x}$, inverse FFT gives the exact derivative.

Finite‑difference: approximate $u_x$ at each interior point by $(u_{i+1,j}-u_{i-1,j})/(2dx)$.

Calculate the $L_2$ error between the two derivative fields; the spectral error is $\\approx 10^{-15}$ (machine precision) while the finite‑difference error is $\\approx 0.12$.

Even on a tiny grid the spectral approach captures the high‑frequency content exactly, which is essential for the fourth‑order dissipation terms.

This prompt supplies only raw spatio‑temporal snapshots and asks the agent to infer the governing model, turning the task into a pure discovery problem.

What does “identifying a mathematical model” entail when the equation is not supplied?

It means selecting a parametric PDE template (e.g., linear terms $\\alpha u$, $\\beta \\nabla^2 u$, fourth‑order dissipation $\\gamma \\nabla^4 u$, and quadratic nonlinearities) and estimating the coefficients so that the simulated evolution matches the observed snapshots within numerical tolerance.

Compute the temporal derivative approximation $\\frac{u^{(1)}-u^{(0)}}{\\Delta t}$ for each grid point.

Compute the Laplacian $\\nabla^2 u^{(0)}$ via spectral multiplication by $-(k_x^2+k_y^2)$.

Set up a linear regression $\\frac{u^{(1)}-u^{(0)}}{\\Delta t} = a\\,u^{(0)} + b\\,\\nabla^2 u^{(0)}$ and solve for $a,b$ (least squares).

Validate the fitted model by predicting $u^{(2)}$ from $u^{(1)}$ and comparing to the true $u^{(2)}$.

Even a minimal linear model can capture the dominant growth/decay rates; residual errors point to missing nonlinear terms.

This prompt adds a list of unrelated PDE families and computational options, testing the agent’s ability to filter irrelevant context and stay focused on the supplied data.

Why should the agent ignore the listed unrelated PDE families and solver options?

Because none of those families are supported by the provided data; attempting to fit, for example, a Cahn–Hilliard model would produce dynamics that do not match the observed spectra, wasting computation and leading to incorrect diagnostics.

Compute $\\nabla^2 u$ spectrally and form the chemical potential $\\mu = \\epsilon^2\\nabla^2 u - f(u)$.

Evaluate $\\nabla^2 \\mu$ and advance $u$ with an explicit Euler step.

Compare the resulting field to the true next snapshot; the mismatch is large (L2 error $>0.5$) because the underlying physics differ.

Recognize the failure and revert to the data‑driven discovery pipeline from B3.

Attempting an unrelated model quickly reveals inconsistency with the data, reinforcing the need to stay within the task’s defined scope.

Compute Costs and Conclusions

Computational cost varies with guidance completeness and cost‑performance trade‑offs differ.

Computational cost is driven not only by the amount of methodological guidance but also by how complete that guidance is.

Removing procedural details (B3, B4) raises token use by 25 %–30 % and time by 18 %–22 % relative to B1, while the most incomplete setting (B2) spikes cost by 59 % in tokens and 32 % in time.

Higher computational budgets do not guarantee better scientific scores; cost‑effective systems can match or exceed more expensive ones.

**Figure 4.** Computational cost under different levels of methodological guidance and cost–performance trade-offs across Agent×Model combinations. (a–b) Average per-task token consumption and execution time under B1–B4. (c) Relationship between per-run monetary cost and B3 scientific score across evaluated systems. Colors denote backbone models, while marker shapes denote agent harnesses.

ASI‑Bench’s B1–B4 structure provides a shared platform for controlled model‑agent comparisons, but its long‑term impact depends on broad community contributions and continual evolution.

Benchmark Development and Authorship

Details the contributors, reviewers, and affiliations behind the ASI‑Bench benchmark.

ASI‑Bench was built collaboratively, with 60 project‑level research tasks spanning 11 scientific domains.

Twenty‑one researchers contributed the retained tasks, and contributors are ordered by how many tasks they kept in the benchmark.

The benchmark underwent five rounds of human review, accumulating over 1,100 task‑review assignments and multiple revisions.

Task contributors include Yuexi Pan, Hengyu Wang, Honghe Ren, Peigan Gao, Jiangyu Zhou, Sijia Chen, Junhao Wu, Huan Wang, Koutian Wu, Cheng Zhang, Yuanning Feng, Qingyuan Zheng, Wenzhe Li, Jiakun Wu, Ruixuan Jia, Junwei Zhou, Ergan Shang, Jingjing Zhou, Yan Xu, Hongrui Zhang, and Liting Mai.

Human reviewers include Jiangyu Zhou, Yuexi Pan, Hengyu Wang, Honghe Ren, Xiaohan Jia, Xueyang Zhou, Cheng Zhang, Yuanning Feng, Sijia Chen, Junhao Wu, Huan Wang, Koutian Wu, Qingyuan Zheng, Peigan Gao, Wenzhe Li, Jiakun Wu, Jingjing Zhou, Ruixuan Jia, Yuexing Hao, Yan Xu, Hongrui Zhang, Zhengxiang Cheng, and Xianglin Ji.

Affiliations represented range from Tsinghua University and MIT to Microsoft Research, AG2 AI, and independent researchers.

Community-Driven Benchmark Evolution

Guidelines for contributing reproducible, scientifically meaningful tasks to the evolving ASI‑Bench benchmark.

ASI‑Bench is deliberately built as an evolving, community‑driven benchmark rather than a static collection of problems. The initial release offers 60 tasks spanning 11 domains, but this is only a fraction of the scientific challenges future AI systems must tackle.

Contributors are asked to submit genuine scientific problems—tasks that require non‑trivial reasoning and produce reproducible artifacts—rather than simple question‑answer pairs. A complete submission must contain a clear objective, the four prompt variants (B1–B4), explicit I/O specifications, a reference‑generation script, an evaluation spec with weighted scoring, required software dependencies, and evidence from local evaluation across all information conditions.

The contribution workflow consists of three stages. First, authors formulate a deterministic, reproducible scientific question and justify its difficulty. Second, they build the B1–B4 prompts, supply a reference generator (e.g., `generate_gt`.py), and define evaluation gates and scorers. Third, they run local tests, submit the package through the online portal, and revise the task based on reviewer feedback.

**Figure 5.** ASI-Bench task contribution portal. The online submission workspace provides a 15-step Guided Flow for constructing and validating new tasks. Representative interfaces show the major stages of task authoring including scientific problem definition, B1–B4 prompt construction, evaluation design with gates and scorers, runtime configuration, task-file preparation, and local testing before review.

Accepted tasks are credited in the benchmark release, reinforcing the view of contributors as co‑builders of ASI‑Bench. By continuously incorporating community‑sourced challenges, the benchmark stays aligned with the frontier of scientific inquiry and helps gauge progress toward artificial superintelligence.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers