IDA-Bench: Evaluating LLMs on Interactive Guided Data Analysis
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915330171863040 |
|---|---|
| author | Li, Hanyu Liu, Haoyu Zhu, Tingyu Guo, Tianyu Zheng, Zeyu Deng, Xiaotie Jordan, Michael I. |
| author_facet | Li, Hanyu Liu, Haoyu Zhu, Tingyu Guo, Tianyu Zheng, Zeyu Deng, Xiaotie Jordan, Michael I. |
| contents | Large Language Models (LLMs) show promise as data analysis agents, but existing benchmarks overlook the iterative nature of the field, where experts' decisions evolve with deeper insights of the dataset. To address this, we introduce IDA-Bench, a novel benchmark evaluating LLM agents in multi-round interactive scenarios. Derived from complex Kaggle notebooks, tasks are presented as sequential natural language instructions by an LLM-simulated user. Agent performance is judged by comparing its final numerical output to the human-derived baseline. Initial results show that even state-of-the-art coding agents (like Claude-3.7-thinking) succeed on < 50% of the tasks, highlighting limitations not evident in single-turn tests. This work underscores the need to improve LLMs' multi-round capabilities for building more reliable data analysis agents, highlighting the necessity of achieving a balance between instruction following and reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_18223 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | IDA-Bench: Evaluating LLMs on Interactive Guided Data Analysis Li, Hanyu Liu, Haoyu Zhu, Tingyu Guo, Tianyu Zheng, Zeyu Deng, Xiaotie Jordan, Michael I. Computation and Language Artificial Intelligence Large Language Models (LLMs) show promise as data analysis agents, but existing benchmarks overlook the iterative nature of the field, where experts' decisions evolve with deeper insights of the dataset. To address this, we introduce IDA-Bench, a novel benchmark evaluating LLM agents in multi-round interactive scenarios. Derived from complex Kaggle notebooks, tasks are presented as sequential natural language instructions by an LLM-simulated user. Agent performance is judged by comparing its final numerical output to the human-derived baseline. Initial results show that even state-of-the-art coding agents (like Claude-3.7-thinking) succeed on < 50% of the tasks, highlighting limitations not evident in single-turn tests. This work underscores the need to improve LLMs' multi-round capabilities for building more reliable data analysis agents, highlighting the necessity of achieving a balance between instruction following and reasoning. |
| title | IDA-Bench: Evaluating LLMs on Interactive Guided Data Analysis |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2505.18223 |