LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909909211152384 |
|---|---|
| author | Ling, Zhan Liu, Kang Yan, Kai Yang, Yifan Lin, Weijian Fan, Ting-Han Shen, Lingfeng Du, Zhengyin Chen, Jiecao |
| author_facet | Ling, Zhan Liu, Kang Yan, Kai Yang, Yifan Lin, Weijian Fan, Ting-Han Shen, Lingfeng Du, Zhengyin Chen, Jiecao |
| contents | Large language models (LLMs) have demonstrated remarkable progress in understanding long-context inputs. However, benchmarks for evaluating the long-context reasoning abilities of LLMs fall behind the pace. Existing benchmarks often focus on a narrow range of tasks or those that do not demand complex reasoning. To address this gap and enable a more comprehensive evaluation of the long-context reasoning capabilities of current LLMs, we propose a new synthetic benchmark, LongReason, which is constructed by synthesizing long-context reasoning questions from a varied set of short-context reasoning questions through context expansion. LongReason consists of 794 multiple-choice reasoning questions with diverse reasoning patterns across three task categories: reading comprehension, logical inference, and mathematical word problems. We evaluate 21 LLMs on LongReason, revealing that most models experience significant performance drops as context length increases. Our further analysis shows that even state-of-the-art LLMs still have significant room for improvement in providing robust reasoning across different tasks. We have open-sourced LongReason under https://huggingface.co/datasets/lz1bytedance/LongReason to support the comprehensive evaluation of LLMs' long-context reasoning capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_15089 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion Ling, Zhan Liu, Kang Yan, Kai Yang, Yifan Lin, Weijian Fan, Ting-Han Shen, Lingfeng Du, Zhengyin Chen, Jiecao Computation and Language Large language models (LLMs) have demonstrated remarkable progress in understanding long-context inputs. However, benchmarks for evaluating the long-context reasoning abilities of LLMs fall behind the pace. Existing benchmarks often focus on a narrow range of tasks or those that do not demand complex reasoning. To address this gap and enable a more comprehensive evaluation of the long-context reasoning capabilities of current LLMs, we propose a new synthetic benchmark, LongReason, which is constructed by synthesizing long-context reasoning questions from a varied set of short-context reasoning questions through context expansion. LongReason consists of 794 multiple-choice reasoning questions with diverse reasoning patterns across three task categories: reading comprehension, logical inference, and mathematical word problems. We evaluate 21 LLMs on LongReason, revealing that most models experience significant performance drops as context length increases. Our further analysis shows that even state-of-the-art LLMs still have significant room for improvement in providing robust reasoning across different tasks. We have open-sourced LongReason under https://huggingface.co/datasets/lz1bytedance/LongReason to support the comprehensive evaluation of LLMs' long-context reasoning capabilities. |
| title | LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2501.15089 |