LongDA: Benchmarking LLM Agents for Long-Document Data Analysis
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915722132717568 |
|---|---|
| author | Li, Yiyang Zhang, Zheyuan Ma, Tianyi Wang, Zehong Murugesan, Keerthiram Zhang, Chuxu Ye, Yanfang |
| author_facet | Li, Yiyang Zhang, Zheyuan Ma, Tianyi Wang, Zehong Murugesan, Keerthiram Zhang, Chuxu Ye, Yanfang |
| contents | We introduce LongDA, a data analysis benchmark for evaluating LLM-based agents under documentation-intensive analytical workflows. In contrast to existing benchmarks that assume well-specified schemas and inputs, LongDA targets real-world settings in which navigating long documentation and complex data is the primary bottleneck. To this end, we manually curate raw data files, long and heterogeneous documentation, and expert-written publications from 17 publicly available U.S. national surveys, from which we extract 505 analytical queries grounded in real analytical practice. Solving these queries requires agents to first retrieve and integrate key information from multiple unstructured documents, before performing multi-step computations and writing executable code, which remains challenging for existing data analysis agents. To support the systematic evaluation under this setting, we develop LongTA, a tool-augmented agent framework that enables document access, retrieval, and code execution, and evaluate a range of proprietary and open-source models. Our experiments reveal substantial performance gaps even among state-of-the-art models, highlighting the challenges researchers should consider before applying LLM agents for decision support in real-world, high-stakes analytical settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_02598 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | LongDA: Benchmarking LLM Agents for Long-Document Data Analysis Li, Yiyang Zhang, Zheyuan Ma, Tianyi Wang, Zehong Murugesan, Keerthiram Zhang, Chuxu Ye, Yanfang Digital Libraries Artificial Intelligence We introduce LongDA, a data analysis benchmark for evaluating LLM-based agents under documentation-intensive analytical workflows. In contrast to existing benchmarks that assume well-specified schemas and inputs, LongDA targets real-world settings in which navigating long documentation and complex data is the primary bottleneck. To this end, we manually curate raw data files, long and heterogeneous documentation, and expert-written publications from 17 publicly available U.S. national surveys, from which we extract 505 analytical queries grounded in real analytical practice. Solving these queries requires agents to first retrieve and integrate key information from multiple unstructured documents, before performing multi-step computations and writing executable code, which remains challenging for existing data analysis agents. To support the systematic evaluation under this setting, we develop LongTA, a tool-augmented agent framework that enables document access, retrieval, and code execution, and evaluate a range of proprietary and open-source models. Our experiments reveal substantial performance gaps even among state-of-the-art models, highlighting the challenges researchers should consider before applying LLM agents for decision support in real-world, high-stakes analytical settings. |
| title | LongDA: Benchmarking LLM Agents for Long-Document Data Analysis |
| topic | Digital Libraries Artificial Intelligence |
| url | https://arxiv.org/abs/2601.02598 |