DataFlow-Harness: A Grounded Code-Agent Platform for Constructing Editable LLM Data Pipelines

Runming He, Zhen Hao Wong, Hao Liang, Zimo Meng, Chengyu Shen, Xiaochen Ma, Wentao Zhang

DataFlow-Harness bridges the NL2Pipeline gap by grounding LLM agents in platform-native DAG construction.

How can we bridge the gap between transient LLM-generated code and persistent, editable data-processing pipelines?

Coding agents often generate disposable scripts that are difficult to audit, edit, or integrate into production data platforms. This disconnect between natural-language intent and persistent, structured workflow assets is the NL2Pipeline gap. DataFlow-Harness replaces free-form script generation with a platform-grounded system that guides agents to build workflows as directed acyclic graphs (DAGs) through typed, incremental mutations. It uses procedural skills and live environment metadata to ensure every change is structurally valid and platform-native. On a 12-task benchmark, the system achieves a 93.3% end-to-end pass rate while reducing construction costs by 72.5% and latency by 49.9% compared to vanilla script-generation baselines.

Paper Primer

The system functions as a mediator between the agent and the data platform. It uses a Request-Validate-Commit protocol: the agent proposes a mutation, the system checks it for structural validity (acyclicity and schema compatibility), and only then commits it to the authoritative backend.

DataFlow-Skills act as a procedural compass for the agent. While the Model Context Protocol (MCP) provides the "what" (the registry of available operators), Skills provide the "how" (recommended assembly sequences and compatibility rules), preventing the agent from constructing semantically incorrect pipelines.

DataFlow-Harness achieves near-baseline reliability with significantly higher efficiency.

On a 12-task benchmark, the system's 93.3% pass rate is within 0.9 percentage points of the Context-Aware Claude Code baseline, while reducing monetary cost by 42.8% and latency by 17.6%. 72.5% cost reduction vs. vanilla script generation.

Grounded pipeline construction improves downstream model performance.

Data produced by DataFlow-Harness pipelines yielded higher average accuracy on math and code benchmarks compared to pipelines authored by unconstrained agents, specifically improving AIME24@32 from 25.1 to 35.9. 2.3 point average improvement across nine general SFT benchmarks.

Why is generating a structured DAG harder for an agent than writing a Python script?

Generating a script allows the agent to rely on its internal parametric knowledge of general-purpose code. Generating a DAG requires the agent to adhere to strict platform-specific constraints, such as operator compatibility and schema matching, which imposes a higher reasoning burden.

Does this system replace the agent's reasoning with hard-coded rules?

No; it constrains the agent's action space. The agent still interprets user intent, but it is guided by DataFlow-Skills and grounded by the MCP layer, ensuring the resulting artifact is a persistent, editable platform object rather than a disposable script.

For data-engineering platforms, this work demonstrates that shifting from "code generation" to "platform-native workflow synthesis" enables both better governance and higher-quality downstream data.

The Pipeline Materialization Gap

Introducing DataFlow‑Harness, which turns transient LLM‑generated scripts into persistent, editable DAG pipelines.

LLM coding agents typically emit free‑form scripts that remain only as transient source files; they are not automatically materialized as persistent, editable platform artifacts. This disconnect—called the NL2Pipeline gap—prevents production teams from inspecting, auditing, and reusing generated workflows.

DataFlow‑Harness steers an LLM to build platform‑native directed acyclic graphs (DAGs) through typed, incremental mutations, guaranteeing that the resulting pipeline lives as a persistent, visual artifact.

The key shift is moving from disposable scripts to persistent, editable DAG artifacts that stay in sync with the LLM’s conversational authoring.

DataFlow-Harness Architecture

The architecture synchronizes agent edits with a visual DAG, guaranteeing a single source of truth.

Agents that generate code often leave the resulting scripts stranded as transient strings, invisible to downstream tooling. This “materialization gap” forces developers to copy‑paste, rewrite, and manually reconcile the artifact. Our system closes that gap by binding every agent mutation to a live, inspectable pipeline representation.

Think of the pipeline as a shared Google Doc: the agent writes edits, the visual editor sees them instantly, and a central server enforces a single, conflict‑free version.

Agent emits mutation 1: add operator B with schema {input: table, output: `cleaned_text`}.

Backend registers B; the DAG now has nodes {A, B} and no edges.

Agent emits mutation 2: add edge (A → B).

Validation checks: (i) no cycle introduced (graph remains a DAG); (ii) A’s output schema (table) matches B’s input schema (table). Both pass.

Commit succeeds; WebSocket pushes the updated DAG to the visual editor, which instantly draws the new edge.

This toy walk‑through shows how a single round‑trip enforces both structural and schema correctness before any user ever sees the change.

How does Pipeline Synchronization differ from simply re‑running the agent after each manual edit?

Re‑running the agent treats the pipeline as a black‑box snapshot, discarding any intermediate edits. Synchronization, by contrast, treats every edit as a first‑class operation that is validated and immediately reflected across all interfaces, preserving continuity and avoiding divergent states.

Data Pipeline Backend is the authoritative store. A pipeline is denoted $P = (D, O, E, S, R)$, where $D$ are data source URIs, $O$ the set of operator instances, $E\subseteq O\times O$ the directed edges, $S$ the input/output schemas, and $R$ runtime metadata such as model endpoints.

DataFlow‑WebUI offers two synchronized modalities. The conversational pane lets users describe goals in natural language, while the visual DAG editor renders $P$ as a directed acyclic graph that can be inspected and edited directly.

Imagine a traffic controller that only lets cars through after checking the light and the road ahead; the layer mediates every pipeline change through a Request‑Validate‑Commit protocol.

Request‑Validate‑Commit loop for a single agent turn.

What would happen if the Validation step were omitted?

Without validation the backend could accept a mutation that creates a cycle or mismatched schemas, causing downstream execution to fail silently or produce corrupted data, and the visual editor would display an impossible graph.

DataFlow‑Skills act like a recipe book that the agent consults before cooking: it supplies step‑by‑step blueprints and compatibility constraints so the agent builds sensible pipelines.

Why can’t the agent rely solely on the operator registry without DataFlow‑Skills?

The registry lists available operators but says nothing about how to combine them safely. Skills provide the missing orchestration knowledge—ensuring that the chosen sequence respects schema flow and domain best practices.

**Figure 1** The DataFlow-Harness architecture. A shared pipeline representation is synchronized across the agent runtime and DataFlow-WebUI. DataFlow-Skills guides construction, while the Validation Engine checks DAG structure and schema compatibility.

**Figure 2.** The dual-modality interface of DATAFLOW-WEBUI, illustrating the synchronization between the conversational agent and the visual DAG editor.

Experimental Framework

Native DAG halves cost while reducing token usage compared to script baselines.

Native DAG (DataFlow‑Harness) reduces total API cost to \$95.5, about 50 % lower than the vanilla script baseline.

Cost: $95.5 for Native DAG vs \$190.7 for Vanilla CC.

The evaluation benchmark comprises twelve tasks across six representative data‑processing pipelines, each defined by a natural‑language objective, sample inputs, and acceptance criteria.

Four agent variants are compared: a vanilla code‑generation baseline, a repository‑aware baseline, a tool‑specification‑only variant (MCP‑only), and the proposed Native DAG approach that synchronizes generated code with a visual DAG editor.

Across all metrics, the Native DAG configuration achieves comparable task success while markedly lowering token consumption and cost, confirming that structured DAG synthesis yields efficiency gains without sacrificing reliability.

Synthesis Effectiveness

DataFlow‑Harness delivers near‑state‑of‑the‑art success while cutting cost and latency.

DataFlow‑Harness achieves $93.3\%$ end‑to‑end pass, improving by $+10.0\%$ over MCP‑Only while staying within $0.9$ points of Context‑Aware CC.

Table 1 reports $93.3\%$ for DataFlow‑Harness and $83.3\%$ for MCP‑Only.

**Table 1.** We report the end-to-end pass rate together with token usage, monetary cost, and generation latency. The pass rate is computed over 120 task runs (12 tasks × 10 trials); efficiency metrics are averaged over the same runs. The best results are highlighted in bold, and the second-best results are underlined.

Workflow synthesis turns a high‑level task description into a concrete, platform‑native DAG that can be inspected, edited, and executed.

How does Workflow Synthesis differ from plain code generation?

Code generation emits a monolithic script that the platform cannot edit; Workflow synthesis produces a structured DAG of native operators, making the artifact inspectable, versionable, and directly executable within the platform.

DataFlow‑Harness attains near‑state‑of‑the‑art success while cutting token usage, cost, and latency dramatically.

Ablation Study

Ablations quantify how each component contributes to pipeline performance.

This section isolates the contribution of each design element by removing it and measuring the resulting performance drop.

The collection of reusable platform operators (PDF parsing, OCR, layout recovery, etc.) that a coding agent can invoke to build data‑processing pipelines.

How does an Operator Ecosystem differ from merely listing available functions?

Listing functions provides names only; the ecosystem also guarantees compatible I/O contracts, standardized error handling, and performance guarantees, so the agent can reliably chain them into a valid DAG.

Procedural guidance raises aggregate end‑to‑end success on knowledge‑heavy tasks from 18/30 to 29/30 runs.

Table 3 shows three procedural‑knowledge‑dependent tasks (1a, 1b, 3b) improving from 6/10 each to perfect 10/10.

On scoring‑task bottlenecks, DataFlow‑Harness eliminates failures, improving from 7/10 to 10/10.

Table 3 entries 4a and 4b show the jump.

Math Pipeline Utility

DataFlow‑Harness yields higher downstream accuracy than a vanilla coding pipeline.

Recall that DataFlow‑Harness synchronizes agent‑generated code with a visual DAG editor, making pipelines inspectable and maintainable.

After one training epoch, pipelines built with DataFlow‑Harness achieve higher downstream accuracy than Vanilla CC.

Table 4 reports 51.6 % average accuracy for DataFlow‑Harness versus 49.9 % for Vanilla CC.

After two training epochs, the advantage of DataFlow‑Harness persists.

Table 4 shows 55.7 % average accuracy for DataFlow‑Harness versus 54.5 % for Vanilla CC.

It measures how useful the data produced by an agent‑authored pipeline is for improving a downstream model’s performance.

How does Downstream Training Utility differ from simply measuring the quantity of generated data?

Utility focuses on the *quality* of the data as reflected in downstream model accuracy, not on raw volume. A pipeline that produces many noisy examples can score low on utility, whereas a smaller but cleaner dataset can yield higher downstream performance.

**Table.** Performance comparison of different training pipelines on various benchmarks.

General SFT Pipeline Utility

DataFlow‑Harness boosts downstream SFT performance, especially on code benchmarks.

The SFT scenario (Prompt 2) asks the agent to synthesize a full instruction‑tuning pipeline from scratch. It proceeds in three stages: topic‑conditioned generation of diverse pairs, a critique‑then‑rewrite refinement, and an LLM‑as‑judge filter.

DataFlow‑Harness raises the average downstream evaluation score by 2.3 points over the vanilla pipeline.

Table 5 shows an average of 63.8 for DataFlow‑Harness versus 61.5 for Vanilla CC across nine benchmarks.

**Table 5.** General SFT pipeline quality via downstream training. Both pipelines are authored by Claude Code from the same from-scratch synthesis prompt (Prompt 2) and generate 10K instruction-response pairs with the same models and API settings. We fine-tune Qwen2.5-7B-Base under an identical recipe (full-parameter SFT, LLaMA-Factory + DeepSpeed ZeRO-3, 8xH20, lr 1e-5, cosine, warmup 0.03, 3 epochs, global batch 128, cutoff 4096, bf16, seed 42) and evaluate with lm-evaluation-harness [8]. Code benchmarks use the EvalPlus [17] variants (HE+/MBPP+). Avg is the mean over all nine benchmarks; best per column in bold.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers