TransientTables: Evaluating LLMs' Reasoning on Temporally Evolving Semi-structured Tables
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913773108854784 |
|---|---|
| author | Shankarampeta, Abhilash Mahajan, Harsh Kataria, Tushar Roth, Dan Gupta, Vivek |
| author_facet | Shankarampeta, Abhilash Mahajan, Harsh Kataria, Tushar Roth, Dan Gupta, Vivek |
| contents | Humans continuously make new discoveries, and understanding temporal sequence of events leading to these breakthroughs is essential for advancing science and society. This ability to reason over time allows us to identify future steps and understand the effects of financial and political decisions on our lives. However, large language models (LLMs) are typically trained on static datasets, limiting their ability to perform effective temporal reasoning. To assess the temporal reasoning capabilities of LLMs, we present the TRANSIENTTABLES dataset, which comprises 3,971 questions derived from over 14,000 tables, spanning 1,238 entities across multiple time periods. We introduce a template-based question-generation pipeline that harnesses LLMs to refine both templates and questions. Additionally, we establish baseline results using state-of-the-art LLMs to create a benchmark. We also introduce novel modeling strategies centered around task decomposition, enhancing LLM performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_01879 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TransientTables: Evaluating LLMs' Reasoning on Temporally Evolving Semi-structured Tables Shankarampeta, Abhilash Mahajan, Harsh Kataria, Tushar Roth, Dan Gupta, Vivek Computation and Language Computer Vision and Pattern Recognition Information Retrieval Humans continuously make new discoveries, and understanding temporal sequence of events leading to these breakthroughs is essential for advancing science and society. This ability to reason over time allows us to identify future steps and understand the effects of financial and political decisions on our lives. However, large language models (LLMs) are typically trained on static datasets, limiting their ability to perform effective temporal reasoning. To assess the temporal reasoning capabilities of LLMs, we present the TRANSIENTTABLES dataset, which comprises 3,971 questions derived from over 14,000 tables, spanning 1,238 entities across multiple time periods. We introduce a template-based question-generation pipeline that harnesses LLMs to refine both templates and questions. Additionally, we establish baseline results using state-of-the-art LLMs to create a benchmark. We also introduce novel modeling strategies centered around task decomposition, enhancing LLM performance. |
| title | TransientTables: Evaluating LLMs' Reasoning on Temporally Evolving Semi-structured Tables |
| topic | Computation and Language Computer Vision and Pattern Recognition Information Retrieval |
| url | https://arxiv.org/abs/2504.01879 |