The Mighty ToRR: A Benchmark for Table Reasoning and Robustness
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912908346130432 |
|---|---|
| author | Ashury-Tahan, Shir Mai, Yifan C, Rajmohan Gera, Ariel Perlitz, Yotam Yehudai, Asaf Bandel, Elron Choshen, Leshem Shnarch, Eyal Liang, Percy Shmueli-Scheuer, Michal |
| author_facet | Ashury-Tahan, Shir Mai, Yifan C, Rajmohan Gera, Ariel Perlitz, Yotam Yehudai, Asaf Bandel, Elron Choshen, Leshem Shnarch, Eyal Liang, Percy Shmueli-Scheuer, Michal |
| contents | Despite its real-world significance, model performance on tabular data remains underexplored, leaving uncertainty about which model to rely on and which prompt configuration to adopt. To address this gap, we create ToRR, a benchmark for Table Reasoning and Robustness, measuring model performance and robustness on table-related tasks. The benchmark includes 10 datasets that cover different types of table reasoning capabilities across varied domains. ToRR goes beyond model performance rankings, and is designed to reflect whether models can handle tabular data consistently and robustly, across a variety of common table representation formats. We present a leaderboard as well as comprehensive analyses of the results of leading models over ToRR. Our results reveal a striking pattern of brittle model behavior, where even strong models are unable to perform robustly on tabular data tasks. Although no specific table format leads to consistently better performance, we show that testing over multiple formats is crucial for reliably estimating model capabilities. Moreover, we show that the reliability boost from testing multiple prompts can be equivalent to adding more test examples. Overall, our findings show that table understanding and reasoning tasks remain a significant challenge. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_19412 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The Mighty ToRR: A Benchmark for Table Reasoning and Robustness Ashury-Tahan, Shir Mai, Yifan C, Rajmohan Gera, Ariel Perlitz, Yotam Yehudai, Asaf Bandel, Elron Choshen, Leshem Shnarch, Eyal Liang, Percy Shmueli-Scheuer, Michal Computation and Language Despite its real-world significance, model performance on tabular data remains underexplored, leaving uncertainty about which model to rely on and which prompt configuration to adopt. To address this gap, we create ToRR, a benchmark for Table Reasoning and Robustness, measuring model performance and robustness on table-related tasks. The benchmark includes 10 datasets that cover different types of table reasoning capabilities across varied domains. ToRR goes beyond model performance rankings, and is designed to reflect whether models can handle tabular data consistently and robustly, across a variety of common table representation formats. We present a leaderboard as well as comprehensive analyses of the results of leading models over ToRR. Our results reveal a striking pattern of brittle model behavior, where even strong models are unable to perform robustly on tabular data tasks. Although no specific table format leads to consistently better performance, we show that testing over multiple formats is crucial for reliably estimating model capabilities. Moreover, we show that the reliability boost from testing multiple prompts can be equivalent to adding more test examples. Overall, our findings show that table understanding and reasoning tasks remain a significant challenge. |
| title | The Mighty ToRR: A Benchmark for Table Reasoning and Robustness |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.19412 |