Researchers at Peking University, Zhongguancun Academy, and Shanghai Institute for Advanced Algorithms Research have released DataFlow-Harness, an open-source framework designed to steer large language model (LLM) agents to construct structured, visually editable data-processing pipelines instead of producing disposable code scripts. The framework aims to make AI-generated workflows persistent, auditable, and easier to integrate into enterprise systems.
Why this matters: the "NL2Pipeline gap"
The authors identify an "NL2Pipeline gap": a disconnect between natural-language workflow specifications and the platform-oriented, structured pipeline assets required in production. Modern coding agents can often generate plausible Python scripts quickly, but those scripts frequently lack grounding in the real platform: they cite unavailable operators, assume outdated dependencies, or fail to produce artifacts that other engineers can inspect, modify, and govern. That gap leads to technical debt and fragile integrations.
How DataFlow-Harness is organized
DataFlow-Harness centers workflow synthesis on four components:
-
Data Pipeline Backend: the authoritative source of truth that represents the pipeline as a directed acyclic graph (DAG) containing data sources, configured pre-built processing modules ("operators"), and execution dependencies. Agents perform "typed mutations" on this backend (e.g., add an operator or connect edges) rather than emitting free-form code.
-
DataFlow-Skills: markdown files that inject domain-specific knowledge into the model's context window. Skills encode compatibility rules, schema inference, and assembly procedures so the model can match data formats and handle complex structures without breaking the pipeline.
-
MCP Tools Layer: a tools layer that exposes the operator registry and the current pipeline state to the AI. The agent proposes structured changes via these tools, and the system validates them to ensure modules are connected in a valid sequence and share compatible data interfaces.
-
DataFlow-WebUI: a dual interface that supports collaboration between humans and AI. Developers can describe requirements in natural language via a conversational interface or inspect and edit the workflow graphically in a DAG editor where AI-proposed changes are visible and can be revised.
The implementation performs static checks against platform metadata before accepting pipeline changes (registered datasets, operators, model-serving references, field flow, parameter usage, and structural validity).
Experimental results: pass rates, costs, latency
The team evaluated DataFlow-Harness on a 12-task benchmark spanning six industrial data-processing scenarios (e.g., QA generation, review governance, schema normalization), using Claude Opus 4.7 as the backbone model.
They compared DataFlow-Harness to three baselines:
- Vanilla CC: unconstrained Claude Code
- Context-Aware CC: an agent with the DataFlow codebase in its context window
- MCP-only: an agent limited to MCP tools and instructed to generate platform-native DAGs without DataFlow-Skills
Key outcomes:
- End-to-end pass rate: DataFlow-Harness achieved 93.3%, which is 10.0 percentage points higher than MCP-only, higher than Vanilla CC (91.7%), and within 0.9 percentage points of Context-Aware CC (94.2%).
- API cost: $0.261 per task, representing a 72.5% reduction compared to Vanilla CC and a 42.8% reduction compared to Context-Aware CC.
- Latency: workflow generation was 49.9% faster than Vanilla CC and 17.6% faster than Context-Aware CC.
DataFlow-Harness was notably effective on complex, domain-heavy tasks. In a textbook-to-VQA extraction example requiring PDF parsing, layout recovery, OCR, figure extraction, multimodal understanding, and long-range QA matching, the system reached 97.2% precision and 87.3% coverage, outperforming baselines by recovering more valid QA pairs through composition of existing platform assets rather than writing new code from scratch.
The framework also handled multi-stage synthetic instruction-data generation pipelines—generating candidate instruction–response pairs, critiquing and rewriting them, scoring them with an LLM-based judge, and filtering out low-quality examples prior to training. In a math data cleaning-and-synthesis task, data produced by DataFlow-Harness led to a model with higher average accuracy on the AIME24 and AIME25 benchmarks than data produced by the vanilla Claude Code pipeline.
Integration tradeoffs and limitations
DataFlow-Harness is released under the Apache 2.0 license, but the current implementation is native to the DataFlow platform rather than a turnkey plugin for Airflow, Prefect, or Spark. Organizations wishing to use other execution backbones must build adapters to connect registries, metadata, and execution interfaces to the agent control layer.
Adoption also requires investment in the boundaries the AI must respect: maintaining an operator registry, defining schemas, and encoding recurring domain procedures as Skills. Because of this overhead, the authors advise against using the framework for small one-off transformations or legacy environments that cannot expose reliable metadata.
The researchers stress that the harness is an engineering control layer—not a substitute for compliance policy, validated detection models, access controls, audit logging, or human approval.
Conclusion
As protocols like MCP standardize, the interaction boundary between engineers and AI agents will shift. The goal of DataFlow-Harness is not to enable autonomous data engineering without oversight, but to improve the division of labor: let agents perform repetitive construction within explicit boundaries while engineers retain responsibility for semantics, policies, and consequential decisions. The project’s source code and documentation are available in the project’s GitHub repository.
