Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909223295647744 |
|---|---|
| author | Fatemi, Bahare Kazemi, Mehran Tsitsulin, Anton Malkan, Karishma Yim, Jinyeong Palowitch, John Seo, Sungyong Halcrow, Jonathan Perozzi, Bryan |
| author_facet | Fatemi, Bahare Kazemi, Mehran Tsitsulin, Anton Malkan, Karishma Yim, Jinyeong Palowitch, John Seo, Sungyong Halcrow, Jonathan Perozzi, Bryan |
| contents | Large language models (LLMs) have showcased remarkable reasoning capabilities, yet they remain susceptible to errors, particularly in temporal reasoning tasks involving complex temporal logic. Existing research has explored LLM performance on temporal reasoning using diverse datasets and benchmarks. However, these studies often rely on real-world data that LLMs may have encountered during pre-training or employ anonymization techniques that can inadvertently introduce factual inconsistencies. In this work, we address these limitations by introducing novel synthetic datasets specifically designed to assess LLM temporal reasoning abilities in various scenarios. The diversity of question types across these datasets enables systematic investigation into the impact of the problem structure, size, question type, fact order, and other factors on LLM performance. Our findings provide valuable insights into the strengths and weaknesses of current LLMs in temporal reasoning tasks. To foster further research in this area, we are open-sourcing the datasets and evaluation framework used in our experiments: https://huggingface.co/datasets/baharef/ToT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_09170 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning Fatemi, Bahare Kazemi, Mehran Tsitsulin, Anton Malkan, Karishma Yim, Jinyeong Palowitch, John Seo, Sungyong Halcrow, Jonathan Perozzi, Bryan Computation and Language Large language models (LLMs) have showcased remarkable reasoning capabilities, yet they remain susceptible to errors, particularly in temporal reasoning tasks involving complex temporal logic. Existing research has explored LLM performance on temporal reasoning using diverse datasets and benchmarks. However, these studies often rely on real-world data that LLMs may have encountered during pre-training or employ anonymization techniques that can inadvertently introduce factual inconsistencies. In this work, we address these limitations by introducing novel synthetic datasets specifically designed to assess LLM temporal reasoning abilities in various scenarios. The diversity of question types across these datasets enables systematic investigation into the impact of the problem structure, size, question type, fact order, and other factors on LLM performance. Our findings provide valuable insights into the strengths and weaknesses of current LLMs in temporal reasoning tasks. To foster further research in this area, we are open-sourcing the datasets and evaluation framework used in our experiments: https://huggingface.co/datasets/baharef/ToT. |
| title | Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.09170 |