AgenticDataBench: A Comprehensive Benchmark for Data Agents

Zhaoyan Sun, Shan Zhong, Daizhou Wen, Jiaxing Han, Guoliang Li, Ying Yan, Peng Zhang, Yu Su, Xiang Qi, Baolin Sun, Chengyuan Yang, Tao Fang, Huaiyu Ruan

AgenticDataBench provides a fine-grained, skill-based benchmark to evaluate and diagnose LLM-driven data science agents.

How do current data-science agents perform across a standardized, skill-based benchmark, and where do they systematically fail?

Data science agents often fail in complex, real-world workflows, but existing benchmarks provide only coarse aggregate scores that obscure why an agent succeeded or failed at specific steps. The authors introduce AgenticDataBench, a benchmark that decomposes data science tasks into a hierarchical set of 433 "skills"—recurring operational patterns like "Handling Missing Data"—to enable fine-grained performance diagnosis. By evaluating state-of-the-art agents across 344 tasks spanning 15 domains, the benchmark reveals that general-purpose agent harnesses currently outperform specialized data-science agents due to superior engineering optimizations and context management.

Paper Primer

The benchmark construction hinges on a hierarchical skill extraction pipeline: it uses large-scale Stack Overflow solutions to identify atomic data operations, then applies embedding-based clustering and LLM-driven refinement to group these into a representative, non-redundant skill hierarchy.

To ensure the benchmark reflects practical complexity, the authors select skill-diverse real-world tasks from fintech business units and generate synthetic tasks for public datasets by sampling realistic "skill paths" from a frequency-weighted skill graph.

General-purpose agent harnesses currently outperform specialized data-science agents.

Performance evaluation across 344 tasks shows that production-grade harnesses like CodeX and Smolagents achieve higher success rates than the specialized DA-Agent. The performance gap is attributed to the specialized agents' lack of robust engineering features, such as advanced data profiling and long-horizon context management.

Why is a "skill-based" approach better than just using standard data science benchmarks?

Standard benchmarks provide only coarse-grained aggregate scores, which hide step-level behaviors; skill-based annotation allows for root-cause analysis by identifying exactly which data-centric operations (e.g., feature engineering vs. modeling) an agent consistently misapplies.

How does the benchmark ensure it covers the breadth of real-world data science?

The benchmark integrates 97 real-world datasets across 15 domains—ranging from financial loan modeling to transportation mobility—and uses a greedy approximation algorithm to select tasks that maximize coverage of the 433 identified data science skills.

Researchers should shift from evaluating agents on final output accuracy alone to diagnosing failure patterns at the skill level, as this benchmark demonstrates that agent success is highly dependent on specific operational competencies.

Introduction and Motivation

Data agents need a fine‑grained benchmark to reveal which data‑science skills they truly master.

Data science extracts actionable insights from heterogeneous raw data, but manual pipelines demand extensive human effort.

LLM‑based data agents promise end‑to‑end automation, yet no benchmark measures their performance with the fine granularity required to diagnose skill‑level strengths and failures.

We need a benchmark that pinpoints which data‑science skills agents master or fail, because coarse aggregate scores hide critical weaknesses.

The paper identifies three concrete challenges: C1 – discovery of highly representative skills; C2 – selection of diverse real‑world tasks; C3 – realistic task generation for public datasets.

AgenticDataBench tackles these challenges by (i) extracting skills from large‑scale Stack Overflow solutions via hierarchical clustering, (ii) selecting task‑solution pairs that maximize skill diversity, and (iii) generating synthetic tasks for uncovered domains using LLM‑guided skill‑composition sampling.

**Figure 1.** Agentic Data Science Benchmark Example.

**Table 1.** *AgenticDataBench* vs Existing Data Agent Benchmarks (– indicates no solution code).

Granular evaluation is essential to diagnose data‑agent strengths and failures across the spectrum of data‑science skills.

Data Science Skill Framework

Defines the skill taxonomy and benchmark construction for data agents.

The preliminaries formalize a hierarchical taxonomy of data‑science operations and describe how we turn it into a benchmark for data agents. This sets the stage for the experimental evaluation that follows.

A skill is a reusable, higher‑level data‑centric operation (e.g., handling missing data) that can be broken into finer‑grained sub‑skills, forming a tree of abstraction.

The seven top‑level categories capture the full lifecycle of a data‑science project.

**Figure 2.** 433 Skills Generated by AgenticDataBench.

Collect 97 real datasets spanning 15 domains and 27.3 GB of data.

Extract recurring operation patterns from 6,510 Stack Overflow solutions and organize them into the hierarchical skill framework.

Generate 344 benchmark tasks by composing skills to reflect realistic data‑science workflows.

Annotate each task with the exact set $S$ of required skills using expert review.

Define an evaluation function that maps agent outputs to a scalar score in $[0,1]$ (e.g., normalized MSE).

Benchmark Construction

How the AgenticDataBench suite is built, from skill extraction to task generation.

This section details the systematic construction of the AgenticDataBench benchmark, describing how skills are extracted, how tasks are chosen for diversity, and how new tasks are generated to achieve full skill coverage.

AgenticDataBench is a benchmark suite that pairs data‑science tasks with explicit skill annotations, enabling fine‑grained evaluation of data agents across a controlled skill space.

How does AgenticDataBench differ from earlier data‑agent benchmarks?

Earlier benchmarks typically offered monolithic tasks without explicit skill breakdowns, making it hard to pinpoint which abilities an agent lacked. AgenticDataBench injects a skill taxonomy, so performance can be traced back to concrete data‑science operations (e.g., “feature engineering” vs. “model evaluation”).

Extract a hierarchical skill set from large‑scale task solutions using a vanilla LLM, embedding‑based clustering, and iterative LLM refinement until the skill count falls below a preset threshold.

Select a diverse subset of real‑world tasks by greedily maximizing uncovered skill coverage under a fixed budget.

Generate additional tasks for under‑represented skill combinations via an LLM‑driven pipeline that samples skill graphs, creates data profiles, and synthesizes task descriptions.

Assemble each benchmark instance with its dataset, ground‑truth solution, skill annotations, and a domain‑specific evaluation function; then feed the instances through the AgenticDataBench execution pipeline (Executor → Data Agent → Leaderboard).

Initially no skills are covered.

Greedy iteration 1: Task B covers two previously uncovered skills (modeling, evaluation) – the most of any task – so it is selected.

Remaining uncovered skills: cleaning, visualization, reporting.

Greedy iteration 2: Task A covers two of the remaining skills (cleaning, visualization) – more than any other candidate – so it is selected.

Result: Selected tasks B and A together cover all five distinct skills.

This toy example shows how the greedy algorithm quickly achieves maximal skill coverage without exhaustive search, which would be NP‑hard.

**Figure 3.** The Workflow of Constructing *AgenticDataBench*.

**Figure 4.** The Overview of AgenticDataBench Pipeline.

Task Generation and Coverage

Describes how tasks are generated using skill coverage to expand the benchmark.

To broaden benchmark coverage we augment the suite with dozens of new public datasets and then synthesize tasks that explicitly target under‑represented skills. The pipeline proceeds in four deterministic stages.

We treat the set of data‑science skills as a weighted graph and sample realistic skill sequences, then ask a language model to turn those sequences into concrete tasks anchored in real datasets.

How does this generation differ from simply picking random skills and datasets?

Random selection ignores the empirical co‑occurrence patterns captured in the skill graph, so the resulting tasks often lack realistic workflow structure. By sampling from the weighted graph we preserve the natural ordering and dependency of skills observed in real solutions.

Skill Graph Creation – aggregate skill traces from existing task solutions, then build a directed graph where nodes are skills and edges encode consecutive usage frequencies.

Task Ingredient Preparation – for each new task, load a domain‑appropriate dataset, sample a skill path from the graph, and retrieve exemplar task‑solution pairs relevant to the sampled skills.

LLM‑based Task Generation – profile the dataset files, synthesize a workflow using the sampled skills, and prompt the LLM to produce a task description that satisfies solvability, skill necessity, and answer verifiability criteria.

Expert Annotation – human annotators validate the pipeline output, refine questions, and author ground‑truth solutions and evaluation functions.

Start node: sample $A$ with probability proportional to $w_A$ → $A$ is chosen.

From $A$, neighbor $B$ is selected with probability proportional to $e_{A\to B}\times w_B = 15\times20$ → $B$ is chosen.

From $B$, neighbor $C$ is selected with probability proportional to $e_{B\to C}\times w_C = 5\times10$ → $C$ is chosen, completing the path $Aarrow Barrow C$.

This concrete path guarantees that the generated task will require loading data, performing a join, and then aggregating, mirroring a realistic data‑science workflow.

**Table 2.** Statistics Across 15 AgenticDataBench Domains.

Experimental Setup

We detail the hardware, execution environment, and models used to benchmark data agents.

The experiments run on a Linux server equipped with 128 GB of RAM and a 3.1 GHz CPU. All data agents are executed inside Docker containers to guarantee safety and consistent evaluation across runs.

We benchmark three leading‑edge LLMs—Qwen3.5‑397B‑A17B, Kimi‑K2.5, and Claude Sonnet 4.6—using each model’s default temperature configuration. These models serve as the language back‑ends for the data‑agent experiments.

Four representative data‑agent harnesses are evaluated; the primary example is DA‑Agent [28], which integrates Bash, Python, and SQL execution tools and reacts to tool feedback in a moving‑window fashion. Table 3 summarizes the skill coverage of these agents, showing both frequently used (TF‑IDF) skills and the most challenging (low‑score) skills across domains.

Evaluation Pipeline

We detail the experimental setup: task categories, agent configurations, domains, and evaluation metrics.

The evaluation pipeline assembles a suite of data‑science tasks, configures agent harnesses, runs agents, and measures performance across multiple skill dimensions.

Define the nine task categories and assign their weight percentages (e.g., Data Manipulation 0.23, Gradient Methods 0.26).

Configure the memory window (80 max steps, 15‑step history, 1‑minute step timeout) and set per‑harness timeouts (Smol 5 min/step, Claude/CodeX 60 min/task).

Select 15 real‑world domains (Financial, Marketing, Transportation, …) and sample tasks from each.

Prepare the benchmark data: 344 tasks, 342 files, 27.3 GB total; compute per‑task averages (6.4 files, 493.4 MB, 23.5 skill applications).

Run each Data Agent on its assigned tasks, respecting the configured step caps and timeouts.

Score outputs with five format‑specific metrics (table, modeling, JSON, chart, text) using Pass@1.

Aggregate skill‑level errors by prompting an LLM to flag misused skills for each task.

Report domain‑wise and skill‑wise results, highlighting representative and challenging skills.

Step 1: Agent generates an initial data‑loading script and executes it (takes 3 min).

Step 2: Agent receives a runtime error, rewrites the script, and re‑executes (4 min).

Step 3: Agent produces a feature‑engineering snippet, runs it (2 min).

Step 4: Agent trains a simple model, evaluates on a validation split (5 min).

Step 5: Agent refines hyperparameters based on validation loss, finishes within the 40‑step cap.

This concrete run shows how the step‑level timeout prevents runaway code execution while the history window keeps the agent’s context manageable.

Overall Performance Results

We report the overall scores, costs, and trade‑offs of the evaluated data agents.

AgenticDataBench evaluates data agents across granular skills; we now present the overall performance.

An Agent Harness is the runtime and tooling that orchestrates a language model’s data‑science actions, handling memory, file I/O, and skill libraries.

CodeX (Kimi‑K2.5) attains the highest total score of 48.8% on AgenticDataBench, surpassing the next best by 3.2%.

Table 4 shows CX (Kimi‑K2.5) with a 48.8% total, the highest among all agent‑harness combinations.

**Table 4.** Scores (%) over AgenticDataBench. SA=Smolagents, DA=DA-Agent, CC=Claude Code, CX=CodeX. ①=Qwen3.5-397B-A17B, ②=Kimi-K2.5, ③=Claude Sonnet 4.6.

**Figure 6.** Trade-off between Cost and Score.

Skill-Level Performance Analysis

Skill scores reveal distinct strengths and weaknesses across agents and categories.

Skill scores vary widely across categories, exposing agent‑specific strengths and weaknesses while still tracking overall task accuracy.

Figure 7 shows median scores ranging from 55 % to 85 % across skill categories, and the ComparisonGrid (below) quantifies per‑agent differences.

All agents are evaluated using the same skill‑annotation pipeline and LLM feedback mechanism, with skill frequencies weighted by $1 - e^{-\text{skill frequency}/40}$ to emphasize common operations.

**Figure 7.** Skill Score Comparison across Categories. SA=Smolagents, DA=DA-Agent, CC=Claude Code, CX=CodeX. ①=Qwen3.5-397B-A17B, ②=Kimi-K2.5, ③=Claude Sonnet 4.6.

**Figure 8.** Skill-Level Strengths and Weaknesses of the Top-Performing Agent per Harness: Codex (Kimi-K2.5), Smolagents (Qwen3.5), Claude Code (Claude 4.6), and DA-Agent (Claude 4.6).

Failure Analysis

We categorize agent errors to reveal skill gaps and system misalignments.

Failure Analysis groups observable errors of a data agent into a fixed taxonomy so we can pinpoint which skill or system component is breaking.

**Figure 9.** Data Agent Failure Distribution. (a) Distribution by Data Agent. SA=Smolagents, DA=DA-Agent, CC=Claude Code, CX=CodeX. ①=Qwen3.5-397B-A17B, ②=Kimi-K2.5, ③=Claude Sonnet 4.6. (b) Distribution by Domain.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers