AssistedDS: Benchmarking How External Domain Knowledge Assists LLMs in Automated Data Science
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917035062067200 |
|---|---|
| author | Luo, An Xian, Xun Du, Jin Tian, Fangqiao Wang, Ganghua Zhong, Ming Zhao, Shengchun Bi, Xuan Liu, Zirui Zhou, Jiawei Srinivasa, Jayanth Kundu, Ashish Fleming, Charles Hong, Mingyi Ding, Jie |
| author_facet | Luo, An Xian, Xun Du, Jin Tian, Fangqiao Wang, Ganghua Zhong, Ming Zhao, Shengchun Bi, Xuan Liu, Zirui Zhou, Jiawei Srinivasa, Jayanth Kundu, Ashish Fleming, Charles Hong, Mingyi Ding, Jie |
| contents | Large language models (LLMs) have advanced the automation of data science workflows. Yet it remains unclear whether they can critically leverage external domain knowledge as human data scientists do in practice. To answer this question, we introduce AssistedDS (Assisted Data Science), a benchmark designed to systematically evaluate how LLMs handle domain knowledge in tabular prediction tasks. AssistedDS features both synthetic datasets with explicitly known generative mechanisms and real-world Kaggle competitions, each accompanied by curated bundles of helpful and adversarial documents. These documents provide domain-specific insights into data cleaning, feature engineering, and model selection. We assess state-of-the-art LLMs on their ability to discern and apply beneficial versus harmful domain knowledge, evaluating submission validity, information recall, and predictive performance. Our results demonstrate three key findings: (1) LLMs frequently exhibit an uncritical adoption of provided information, significantly impairing their predictive performance when adversarial content is introduced, (2) helpful guidance is often insufficient to counteract the negative influence of adversarial information, and (3) in Kaggle datasets, LLMs often make errors in handling time-series data, applying consistent feature engineering across different folds, and interpreting categorical variables correctly. These findings highlight a substantial gap in current models' ability to critically evaluate and leverage expert knowledge, underscoring an essential research direction for developing more robust, knowledge-aware automated data science systems. Our data and code are publicly available here: https://github.com/jeremyxianx/Assisted-DS |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_13992 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AssistedDS: Benchmarking How External Domain Knowledge Assists LLMs in Automated Data Science Luo, An Xian, Xun Du, Jin Tian, Fangqiao Wang, Ganghua Zhong, Ming Zhao, Shengchun Bi, Xuan Liu, Zirui Zhou, Jiawei Srinivasa, Jayanth Kundu, Ashish Fleming, Charles Hong, Mingyi Ding, Jie Machine Learning Artificial Intelligence Computation and Language Methodology 62-07, 62-08, 68T05, 68T07, 68T01, 68T50 I.2.0; I.2.6; I.2.7; I.5.1; I.5.4; H.2.8; G.3 Large language models (LLMs) have advanced the automation of data science workflows. Yet it remains unclear whether they can critically leverage external domain knowledge as human data scientists do in practice. To answer this question, we introduce AssistedDS (Assisted Data Science), a benchmark designed to systematically evaluate how LLMs handle domain knowledge in tabular prediction tasks. AssistedDS features both synthetic datasets with explicitly known generative mechanisms and real-world Kaggle competitions, each accompanied by curated bundles of helpful and adversarial documents. These documents provide domain-specific insights into data cleaning, feature engineering, and model selection. We assess state-of-the-art LLMs on their ability to discern and apply beneficial versus harmful domain knowledge, evaluating submission validity, information recall, and predictive performance. Our results demonstrate three key findings: (1) LLMs frequently exhibit an uncritical adoption of provided information, significantly impairing their predictive performance when adversarial content is introduced, (2) helpful guidance is often insufficient to counteract the negative influence of adversarial information, and (3) in Kaggle datasets, LLMs often make errors in handling time-series data, applying consistent feature engineering across different folds, and interpreting categorical variables correctly. These findings highlight a substantial gap in current models' ability to critically evaluate and leverage expert knowledge, underscoring an essential research direction for developing more robust, knowledge-aware automated data science systems. Our data and code are publicly available here: https://github.com/jeremyxianx/Assisted-DS |
| title | AssistedDS: Benchmarking How External Domain Knowledge Assists LLMs in Automated Data Science |
| topic | Machine Learning Artificial Intelligence Computation and Language Methodology 62-07, 62-08, 68T05, 68T07, 68T01, 68T50 I.2.0; I.2.6; I.2.7; I.5.1; I.5.4; H.2.8; G.3 |
| url | https://arxiv.org/abs/2506.13992 |