RADAR: Benchmarking Language Models on Imperfect Tabular Data
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911241958588416 |
|---|---|
| author | Gu, Ken Zhang, Zhihan Lin, Kate Zhang, Yuwei Paruchuri, Akshay Yu, Hong Kazemi, Mehran Ayush, Kumar Heydari, A. Ali Xu, Maxwell A. Narayanswamy, Girish Liu, Yun Poh, Ming-Zher Yang, Yuzhe Malhotra, Mark Patel, Shwetak Palangi, Hamid Xu, Xuhai McDuff, Daniel Althoff, Tim Liu, Xin |
| author_facet | Gu, Ken Zhang, Zhihan Lin, Kate Zhang, Yuwei Paruchuri, Akshay Yu, Hong Kazemi, Mehran Ayush, Kumar Heydari, A. Ali Xu, Maxwell A. Narayanswamy, Girish Liu, Yun Poh, Ming-Zher Yang, Yuzhe Malhotra, Mark Patel, Shwetak Palangi, Hamid Xu, Xuhai McDuff, Daniel Althoff, Tim Liu, Xin |
| contents | Language models (LMs) are increasingly being deployed to perform autonomous data analyses. However, their data awareness -- the ability to recognize, reason over, and appropriately handle data artifacts such as missing values, outliers, and logical inconsistencies -- remains underexplored. These artifacts are especially common in real-world tabular data and, if mishandled, can significantly compromise the validity of analytical conclusions. To address this gap, we present RADAR, a benchmark for systematically evaluating data-aware reasoning on tabular data. We develop a framework to simulate data artifacts via programmatic perturbations to enable targeted evaluation of model behavior. RADAR comprises 2980 table query pairs, grounded in real-world data spanning 9 domains and 5 data artifact types. In addition to evaluating artifact handling, RADAR systematically varies table size to study how reasoning performance holds when increasing table size. Our evaluation reveals that, despite decent performance on tables without data artifacts, frontier models degrade significantly when data artifacts are introduced, exposing critical gaps in their capacity for robust, data-aware analysis. Designed to be flexible and extensible, RADAR supports diverse perturbation types and controllable table sizes, offering a valuable resource for advancing tabular reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_08249 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RADAR: Benchmarking Language Models on Imperfect Tabular Data Gu, Ken Zhang, Zhihan Lin, Kate Zhang, Yuwei Paruchuri, Akshay Yu, Hong Kazemi, Mehran Ayush, Kumar Heydari, A. Ali Xu, Maxwell A. Narayanswamy, Girish Liu, Yun Poh, Ming-Zher Yang, Yuzhe Malhotra, Mark Patel, Shwetak Palangi, Hamid Xu, Xuhai McDuff, Daniel Althoff, Tim Liu, Xin Databases Computation and Language Language models (LMs) are increasingly being deployed to perform autonomous data analyses. However, their data awareness -- the ability to recognize, reason over, and appropriately handle data artifacts such as missing values, outliers, and logical inconsistencies -- remains underexplored. These artifacts are especially common in real-world tabular data and, if mishandled, can significantly compromise the validity of analytical conclusions. To address this gap, we present RADAR, a benchmark for systematically evaluating data-aware reasoning on tabular data. We develop a framework to simulate data artifacts via programmatic perturbations to enable targeted evaluation of model behavior. RADAR comprises 2980 table query pairs, grounded in real-world data spanning 9 domains and 5 data artifact types. In addition to evaluating artifact handling, RADAR systematically varies table size to study how reasoning performance holds when increasing table size. Our evaluation reveals that, despite decent performance on tables without data artifacts, frontier models degrade significantly when data artifacts are introduced, exposing critical gaps in their capacity for robust, data-aware analysis. Designed to be flexible and extensible, RADAR supports diverse perturbation types and controllable table sizes, offering a valuable resource for advancing tabular reasoning. |
| title | RADAR: Benchmarking Language Models on Imperfect Tabular Data |
| topic | Databases Computation and Language |
| url | https://arxiv.org/abs/2506.08249 |