Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2402.17168 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909121193705472 |
|---|---|
| author | Zhang, Yuge Jiang, Qiyang Han, Xingyu Chen, Nan Yang, Yuqing Ren, Kan |
| author_facet | Zhang, Yuge Jiang, Qiyang Han, Xingyu Chen, Nan Yang, Yuqing Ren, Kan |
| contents | In the era of data-driven decision-making, the complexity of data analysis necessitates advanced expertise and tools of data science, presenting significant challenges even for specialists. Large Language Models (LLMs) have emerged as promising aids as data science agents, assisting humans in data analysis and processing. Yet their practical efficacy remains constrained by the varied demands of real-world applications and complicated analytical process. In this paper, we introduce DSEval -- a novel evaluation paradigm, as well as a series of innovative benchmarks tailored for assessing the performance of these agents throughout the entire data science lifecycle. Incorporating a novel bootstrapped annotation method, we streamline dataset preparation, improve the evaluation coverage, and expand benchmarking comprehensiveness. Our findings uncover prevalent obstacles and provide critical insights to inform future advancements in the field. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_17168 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Benchmarking Data Science Agents Zhang, Yuge Jiang, Qiyang Han, Xingyu Chen, Nan Yang, Yuqing Ren, Kan Artificial Intelligence Computation and Language In the era of data-driven decision-making, the complexity of data analysis necessitates advanced expertise and tools of data science, presenting significant challenges even for specialists. Large Language Models (LLMs) have emerged as promising aids as data science agents, assisting humans in data analysis and processing. Yet their practical efficacy remains constrained by the varied demands of real-world applications and complicated analytical process. In this paper, we introduce DSEval -- a novel evaluation paradigm, as well as a series of innovative benchmarks tailored for assessing the performance of these agents throughout the entire data science lifecycle. Incorporating a novel bootstrapped annotation method, we streamline dataset preparation, improve the evaluation coverage, and expand benchmarking comprehensiveness. Our findings uncover prevalent obstacles and provide critical insights to inform future advancements in the field. |
| title | Benchmarking Data Science Agents |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2402.17168 |