DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915653237080064 |
|---|---|
| author | Lei, Fangyu Meng, Jinxiang Huang, Yiming Zhao, Junjie Zhang, Yitong Luo, Jianwen Zou, Xin Yang, Ruiyi Shi, Wenbo Gao, Yan He, Shizhu Wang, Zuo Liu, Qian Wang, Yang Wang, Ke Zhao, Jun Liu, Kang |
| author_facet | Lei, Fangyu Meng, Jinxiang Huang, Yiming Zhao, Junjie Zhang, Yitong Luo, Jianwen Zou, Xin Yang, Ruiyi Shi, Wenbo Gao, Yan He, Shizhu Wang, Zuo Liu, Qian Wang, Yang Wang, Ke Zhao, Jun Liu, Kang |
| contents | Real-world enterprise data intelligence workflows encompass data engineering that turns raw sources into analytical-ready tables and data analysis that convert those tables into decision-oriented insights. We introduce DAComp, a benchmark of 210 tasks that mirrors these complex workflows. Data engineering (DE) tasks require repository-level engineering on industrial schemas, including designing and building multi-stage SQL pipelines from scratch and evolving existing systems under evolving requirements. Data analysis (DA) tasks pose open-ended business problems that demand strategic planning, exploratory analysis through iterative coding, interpretation of intermediate results, and the synthesis of actionable recommendations. Engineering tasks are scored through execution-based, multi-metric evaluation. Open-ended tasks are assessed by a reliable, experimentally validated LLM-judge, which is guided by hierarchical, meticulously crafted rubrics. Our experiments reveal that even state-of-the-art agents falter on DAComp. Performance on DE tasks is particularly low, with success rates under 20%, exposing a critical bottleneck in holistic pipeline orchestration, not merely code generation. Scores on DA tasks also average below 40%, highlighting profound deficiencies in open-ended reasoning and demonstrating that engineering and analysis are distinct capabilities. By clearly diagnosing these limitations, DAComp provides a rigorous and realistic testbed to drive the development of truly capable autonomous data agents for enterprise settings. Our data and code are available at https://da-comp.github.io |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_04324 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle Lei, Fangyu Meng, Jinxiang Huang, Yiming Zhao, Junjie Zhang, Yitong Luo, Jianwen Zou, Xin Yang, Ruiyi Shi, Wenbo Gao, Yan He, Shizhu Wang, Zuo Liu, Qian Wang, Yang Wang, Ke Zhao, Jun Liu, Kang Computation and Language Artificial Intelligence Real-world enterprise data intelligence workflows encompass data engineering that turns raw sources into analytical-ready tables and data analysis that convert those tables into decision-oriented insights. We introduce DAComp, a benchmark of 210 tasks that mirrors these complex workflows. Data engineering (DE) tasks require repository-level engineering on industrial schemas, including designing and building multi-stage SQL pipelines from scratch and evolving existing systems under evolving requirements. Data analysis (DA) tasks pose open-ended business problems that demand strategic planning, exploratory analysis through iterative coding, interpretation of intermediate results, and the synthesis of actionable recommendations. Engineering tasks are scored through execution-based, multi-metric evaluation. Open-ended tasks are assessed by a reliable, experimentally validated LLM-judge, which is guided by hierarchical, meticulously crafted rubrics. Our experiments reveal that even state-of-the-art agents falter on DAComp. Performance on DE tasks is particularly low, with success rates under 20%, exposing a critical bottleneck in holistic pipeline orchestration, not merely code generation. Scores on DA tasks also average below 40%, highlighting profound deficiencies in open-ended reasoning and demonstrating that engineering and analysis are distinct capabilities. By clearly diagnosing these limitations, DAComp provides a rigorous and realistic testbed to drive the development of truly capable autonomous data agents for enterprise settings. Our data and code are available at https://da-comp.github.io |
| title | DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2512.04324 |