DataPrep-Bench: Benchmarking LLMs as Training Data Preparators

Hao Liang, Qifeng Cai, Yibo Lin, Jianzhuo Du, Qifeng Xia, Sizhe Qiu, Linzhuang Sun, Meiyi Qiang, Zhaoyang Han, Xiaochen Ma, Bohan Zeng, Ruichuan An, Conghui He, Wentao Zhang

DataPrep-Bench provides a unified, downstream-grounded framework to evaluate how LLMs construct training data and predict its quality.

How can we systematically measure the effectiveness of LLMs and agents in preparing high-quality training data for downstream model fine-tuning?

LLM-driven data preparation is currently an anecdotal field where researchers use disparate sources and models, making it impossible to tell which construction strategies actually yield useful training data or which quality metrics are predictive. The authors introduce DataPrep-Bench, a unified benchmark that evaluates both data construction and quality evaluation under a shared protocol, using identical raw sources, base models, and downstream benchmarks. The benchmark reveals that adding synthetic domain data often hurts performance, and it introduces two baselines: a skill-guided agent for construction and a distribution-based metric for quality evaluation.

Paper Primer

Data-Construction-Skill (DCS) operationalizes construction by moving task semantics, output schemas, and validation rules into a reusable skill layer rather than a one-off prompt. The agent acts as a planner that iterates over source chunks, while the skill layer acts as a control interface that enforces faithfulness, diversity, and schema consistency.

The Distributional Alignment Score (DAS) measures the Maximum Mean Discrepancy (MMD) between a candidate dataset and a domain proxy. It treats the candidate and proxy as feature distributions in a kernel space: DAS is the negative MMD distance, where higher scores indicate stronger alignment with the target domain's distribution.

DAS is the most reliable quality-evaluation metric across domains.

It is the only metric that simultaneously clears Pearson correlation r > 0.70 in Math, Science, and Medical domains. Outperforms existing quality-, diversity-, and heuristic-based evaluators.

Data-Construction-Skill significantly improves domain-specific knowledge extraction.

Lifts the Dolly-only baseline by nearly 20 points absolute on Llama-3.1-8B Finance. Competitive with the strongest agent- and DataFlow-based methods.

Why does the benchmark require a "downstream-grounded" protocol?

Existing metrics often rely on surface-level properties like fluency or diversity, which fail to predict whether a dataset will actually improve a model's performance on a specific task. Grounding in downstream performance ensures that a "high quality" score corresponds to actual training utility.

What is the primary limitation of current data construction methods identified by the benchmark?

The authors observe that adding synthetic domain data on top of a general instruction-following corpus (Dolly-15k) often degrades downstream performance. This suggests that current generators struggle to produce data that is both faithful to the source and useful for model training.

The Need for Data-Centric Benchmarking

Introducing DataPrep‑Bench, a unified benchmark for LLM‑driven data construction and quality evaluation.

The quality of training data fundamentally determines what large language models can do, yet the community lacks a single benchmark that measures how well LLMs, agents, and data‑centric pipelines actually prepare that data.

We view LLM‑driven data preparation as two complementary capabilities: *data construction*, which turns raw sources into supervised training examples, and *data quality evaluation*, which predicts the downstream utility of candidate datasets before any fine‑tuning.

A benchmark that simultaneously judges how well LLMs can build training data and how accurately a metric can forecast that data’s downstream value, using the same downstream models as the gold standard.

In the **Data Construction Track**, each method consumes the same raw domain sources and outputs a supervised dataset; we fine‑tune a base model on that dataset together with the shared Dolly‑15k corpus and evaluate on held‑out domain benchmarks.

The **Data Quality Evaluation Track** supplies a pool of candidate datasets to a scoring function; we then measure how linearly the scores predict downstream fine‑tuning results, releasing the Distributional Alignment Score (DAS) which computes an MMD‑based distance to a domain proxy.

(T1) Adding synthesized domain data on top of Dolly‑15k often degrades performance, exposing a gap that surface‑level quality proxies miss.

**Figure 1.** The overall framework of DataPrep-Bench.

The key shift is from evaluating models themselves to evaluating the data that fuels them.

Prior Approaches to Data Construction

We survey LLM-driven data construction, quality evaluation, and existing benchmarks to contextualize our framework.

LLM‑based data construction has become a standard technique for building training and post‑training corpora. Existing methods fall into three groups: seed‑based synthesis (e.g., Self‑Instruct, WizardLM, UltraChat), domain‑grounded corpora that use authoritative sources (e.g., UltraMedical, MegaScience), and workflow‑style pipelines that decompose construction into reusable stages such as chunking, filtering, and verification (e.g., DataFlow, Text2SQL‑Flow, step‑level verification).

Evaluating data quality for LLM training spans per‑sample scorers—QuRating, Deita‑Quality, Deita‑Complexity, FineWeb‑Edu, PairQual—and model‑side signals like Superfiltering, LESS, or perplexity‑based metrics. At the dataset level, diversity estimators such as Vendi Score (with BERT‑Vendi or SimCSE‑Vendi variants), lexical measures (MTLD, HD‑D, n‑gram distinctness), embedding‑based Task2Vec, and distribution‑alignment techniques (e.g., Maximum Mean Discrepancy) provide holistic assessments.

Several benchmarks assess data‑centric preparation strategies via downstream performance, including DataComp/DataComp‑LM, DataPerf, and DCBENCH. While systems like Data‑Juicer and DataFlow offer modular tooling, they lack a unified evaluation of LLM‑driven data preparation. DataPrep‑Bench fills this gap as the first benchmark that directly measures how well LLMs perform data construction and quality evaluation across domains and model sizes.

Benchmark Architecture and Tracks

Defines the two benchmark tracks, their inputs, and how performance is measured.

This track asks whether a method can turn raw domain books into a supervised fine‑tuning dataset that actually improves a downstream benchmark.

How does this construction track differ from simply fine‑tuning on the raw books directly?

Fine‑tuning on raw books would treat whole documents as training examples, which yields poor signal for instruction following. The track requires a method to synthesize explicit question‑answer pairs, turning unstructured knowledge into the supervised format that LLMs actually learn from.

This track measures whether a scoring function can predict, before any training, which candidate datasets will yield higher downstream performance.

Why not use a rank‑based correlation (Spearman) instead of Pearson?

Spearman would ignore the actual spacing between downstream scores; the benchmark expects a metric to reflect that a candidate yielding 78 % accuracy is materially better than one yielding 70 %, not just that it ranks higher.

Collect domain‑specific Markdown folders $B_k$ (books, long‑form texts) and candidate pools $X_k$ (public SFT corpora).

For the Data Construction Track, run a construction method $M$ on $B_k$ to produce $X^{(M)}$.

Fine‑tune the base model $f_0$ on each dataset (either $X^{(M)}$ or each $X_{k,i}$ in the quality track) using the shared SFT protocol.

Evaluate the adapted model on the downstream benchmark $T_k$ to obtain $\text{Perf}$ scores.

For the Quality Evaluation Track, compute metric scores $q_{k,i}=Q(X_{k,i})$ before fine‑tuning.

Calculate Pearson correlation $\rho(Q,D_k)$ between $q_k$ and the collected $s_k$ scores.

Convert $b_{1},b_{2}$ to Markdown; the pipeline reads 2 files.

Prompt the LLM to generate QA pairs; it returns three (q,a) tuples.

Form the SFT dataset $X^{(M)}=\{(q_1,a_1),(q_2,a_2),(q_3,a_3)\}$.

Run the shared SFT routine on $f_0$ for 200 steps; training completes in 5 minutes.

Evaluate the fine‑tuned model on $T_{\text{Math}}$; the reported accuracy is 78.2 %.

This tiny end‑to‑end run shows how a method’s output size (3 pairs) directly translates into a measurable downstream gain, illustrating the concrete link between construction and performance.

**Table 1.** Dataset categorization by domain and usage type.

Data Construction Baselines

We detail the skill‑guided construction pipeline and the distributional alignment metric used as baselines.

This section spells out the two baseline methods that power the DataPrep‑Bench tracks: a skill‑guided agentic pipeline for building QA‑style supervision (Data‑Construction‑Skill) and a distribution‑based metric for estimating downstream utility (Distributional Alignment Score).

We give an LLM a reusable “skill” that tells it what counts as a valid QA pair, then let the model plan over a long book, chunk by chunk, to produce those pairs automatically.

Chunk 1 (Paragraph 1) is retained; the agent extracts the definition and emits $G_{\text{concept}}$ QA: “What is gravity?” → “The force that pulls objects toward Earth.”

Chunk 2 (Paragraph 2) yields a reasoning QA in $G_{\text{reason}}$: “Why does the ball accelerate?” → “Because gravity exerts a constant force, producing constant acceleration.”

The same chunk also produces a case QA in $G_{\text{case}}$: “If you drop a ball from 2 m, how long does it take to hit the ground?” → “Approximately 0.64 s (using $t=\sqrt{2h/g}$).”

All three QA pairs are collected; the final $X^{\text{MDCS}}$ contains three entries, each self‑contained and free of document‑relative references.

The skill guarantees that each generated QA follows a prescribed schema (question, answer, self‑contained) and that invalid patterns are filtered out automatically.

How does Data‑Construction‑Skill differ from a single “one‑shot” prompt that asks an LLM to generate QA pairs?

One‑shot prompting treats the whole book as a monolithic input and cannot enforce consistency, coverage, or validation. Data‑Construction‑Skill breaks the book into chunks, supplies a reusable schema, and lets the agent plan, filter, and resume, which yields higher fidelity and scalability.

DAS measures how closely a candidate training set’s feature distribution matches a domain‑specific proxy, using a kernel‑based distance that requires no downstream fine‑tuning.

Encode every sample in the candidate dataset $X_{k,i}$ and in the proxy $X_{\text{proxy}}$ with the fixed encoder $\phi$.

Compute the empirical MMD between the two feature sets using the Gaussian RBF kernel (Eq. 12).

Return the negative MMD as the DAS value (Eq. 16).

Why not compute MMD directly against the benchmark test set instead of a proxy?

Using the test set would leak evaluation data into the metric, contaminating the benchmark. The proxy approximates the target distribution without overlap, preserving a clean evaluation while still providing a theoretically motivated signal.

**Figure 2.** Data construction prompt used to guide the LLM.

**Figure 3.** Data construction prompt used to guide the agent.

Experimental Overview

We compare data‑construction and quality‑evaluation tracks under identical conditions.

All construction methods are evaluated with the same base model, raw corpus, and downstream benchmarks, eliminating data‑source variance.

Both tracks keep the raw source corpus (Section 3.2) and benchmark suite (Section 3.5) fixed while varying only the synthetic generator.

Across all experiments the raw source corpus from Section 3.2 and the downstream benchmark suite from Section 3.5 remain unchanged; likewise, the candidate pools and proxy datasets described in Section 3.4 are shared by every quality‑metric evaluation.

Construction Evaluation Protocol

Agent‑based generators boost downstream fine‑tuning performance while keeping all other settings fixed.

It is the accuracy (or other task‑specific score) a model attains after being fine‑tuned on a synthetic dataset and then evaluated on real downstream benchmarks.

Agent‑based generators substantially improve downstream fine‑tuning performance compared to direct LLM‑based generators.

Table 3 shows the Skill (Claude Opus 4.6) variant reaches 25.2 % on GSM8K, whereas the direct LLM baseline (Claude Opus 4.6) scores 0.0 %.

All generators consume the identical raw source corpus, and every fine‑tuning run uses the same LlamaFactory hyperparameters on the two base models (Qwen2.5‑7B and Llama‑3.1‑8B) jointly with Dolly‑15k.

Quality Metric Baselines

Evaluating data quality metrics and their impact on downstream performance.

We evaluate 18 quality and diversity operators—including the new DAS metric—on the multi‑domain candidate pools. All experiments share the same fine‑tuning recipe, base model, and Dolly‑15k‑plus synthetic data setup.

They measure how closely a candidate dataset mirrors the target domain’s distribution while also rewarding internal diversity.

How does a DataFlow Metric differ from a traditional perplexity‑based quality score?

Perplexity only gauges how predictable a token sequence is under a language model, ignoring diversity. DataFlow combines a relevance/quality component with an explicit diversity term, so a dataset that is easy to predict but homogeneous will score lower than a diverse, high‑quality set.

DataFlow‑guided synthetic data yields higher downstream accuracy than the Dolly‑15k‑only baseline.

Table 4 shows the average accuracy across 13 benchmarks rises from 58.2 % (baseline) to 70.2 % with DataFlow selection.

All other factors—model architecture, fine‑tuning hyperparameters, and the Dolly‑15k corpus—were held constant across rows, isolating the effect of the data quality metric.

Data Construction Performance

Skill‑guided construction boosts Finance scores while synthetic data often harms performance.

Skill‑guided construction raises Finance performance by roughly 20 points over the Dolly‑15k baseline.

Table 5 shows the Skill (Claude Opus 4.6) generator achieving a Finance average of 84.2 versus 9.9 for Dolly‑15k only.

All experimental conditions share the same instruction‑following corpus (Dolly‑15k) and differ only by the addition of domain‑specific synthetic data.

**Table 1.** Performance comparison of different training data generators across Math, General, and Finance benchmarks.

**Table 3.** Performance of Llama-3.1-8B on Math, General, and Finance benchmarks after fine-tuning on datasets synthesized by different generators. All rows are fine-tuned on Dolly-15k (a general instruction-following corpus) jointly with the domain-specific dataset produced by the named generator; † marks the Dolly-15k only reference baseline, fine-tuned on Dolly-15k alone without any domain-specific synthetic data. Benchmark abbreviations: MM = MinervaMath, OB = OlympiadBench, M500 = MATH-500, GK24 = Gaokao2024, MR = MMLU-Redux, CKQA = CPA-KQA, FEKR = FineEval-KR, XFB = XFinBench.

**Table.** Performance of various training data generators across Law, Medical, and Science domains.

**Table.** Performance comparison of different Training Data Generators across Law, Medical, and Science domains.

Predictive Power of Metrics

DataPrep‑Bench gauges downstream gains and metric predictiveness for LLM‑driven data pipelines.

DAS is the strongest predictor of downstream performance across reasoning‑intensive domains, leading or tying in four of six domains and exceeding $r$ > 0.70 in Math, Science, and Medical.

Observed in Tables 6 and 7, where DAS attains the highest average Pearson correlations in the cited domains.

All metrics were evaluated on the same three base models and the same candidate pools per domain, ensuring that differences arise solely from the scoring functions themselves.

**Table 6.** Pearson correlation between data quality metric and downstream performance on the General Text, Math, and Science domains, along with computational overhead. Time (s) values represent the average processing duration per candidate dataset across all six domains of DataPrep-Bench (excluding model loading and target set embedding costs). Correlation values are reported with two decimal places. Light yellow and light green indicate positive and negative correlations, respectively. Underlined values in model columns indicate that the correlation is consistent with the theoretical expectation and reaches statistical significance (p < 0.05). In each domain, the strongest consistent average correlation in the Avg column is also bolded.

**Table.** Comparison of various metrics across Medical, Finance, and Law domains using Qwen, Llama-8B, and Mistral models.

Benchmark Details and Supplemental Data

Appendix details the benchmark suite, implementation settings, and prompt templates used in the study.

The appendix enumerates the full set of evaluation benchmarks spanning general text, mathematics, science, medicine, finance, and law. Each benchmark is described with its scope and intended reasoning challenge.

Training hyperparameters for all data‑construction experiments use a uniform 3‑epoch cosine schedule with an initial learning rate of $5 \times 10^{-6}$ and a global batch size of 32. Dataset size design leaves sample counts unconstrained, allowing each method to emit as many examples as its pipeline produces, which makes yield an intrinsic efficiency metric.

**Table.** Dataset size distribution across different generator types and domains.

The appendix also provides the exact prompts used for LLM‑based and agent‑based generators, including the JSON schema for QA pairs and the skill‑driven constraints for autonomous data construction. These templates ensure reproducibility and enforce strict adherence to the source markdown files.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers