Chain-of-Thought Matters: Improving Long-Context Language Models with Reasoning Path Supervision
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913712607068160 |
|---|---|
| author | Zhu, Dawei Wei, Xiyu Zhao, Guangxiang Wu, Wenhao Zou, Haosheng Ran, Junfeng Wang, Xun Sun, Lin Zhang, Xiangzheng Li, Sujian |
| author_facet | Zhu, Dawei Wei, Xiyu Zhao, Guangxiang Wu, Wenhao Zou, Haosheng Ran, Junfeng Wang, Xun Sun, Lin Zhang, Xiangzheng Li, Sujian |
| contents | Recent advances in Large Language Models (LLMs) have highlighted the challenge of handling long-context tasks, where models need to reason over extensive input contexts to aggregate target information. While Chain-of-Thought (CoT) prompting has shown promise for multi-step reasoning, its effectiveness for long-context scenarios remains underexplored. Through systematic investigation across diverse tasks, we demonstrate that CoT's benefits generalize across most long-context scenarios and amplify with increasing context length. Motivated by this critical observation, we propose LongRePS, a process-supervised framework that teaches models to generate high-quality reasoning paths for enhanced long-context performance. Our framework incorporates a self-sampling mechanism to bootstrap reasoning paths and a novel quality assessment protocol specifically designed for long-context scenarios. Experimental results on various long-context benchmarks demonstrate the effectiveness of our approach, achieving significant improvements over outcome supervision baselines on both in-domain tasks (+13.6/+3.8 points for LLaMA/Qwen on MuSiQue) and cross-domain generalization (+9.3/+8.1 points on average across diverse QA tasks). Our code, data and trained models are made public to facilitate future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_20790 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Chain-of-Thought Matters: Improving Long-Context Language Models with Reasoning Path Supervision Zhu, Dawei Wei, Xiyu Zhao, Guangxiang Wu, Wenhao Zou, Haosheng Ran, Junfeng Wang, Xun Sun, Lin Zhang, Xiangzheng Li, Sujian Computation and Language Recent advances in Large Language Models (LLMs) have highlighted the challenge of handling long-context tasks, where models need to reason over extensive input contexts to aggregate target information. While Chain-of-Thought (CoT) prompting has shown promise for multi-step reasoning, its effectiveness for long-context scenarios remains underexplored. Through systematic investigation across diverse tasks, we demonstrate that CoT's benefits generalize across most long-context scenarios and amplify with increasing context length. Motivated by this critical observation, we propose LongRePS, a process-supervised framework that teaches models to generate high-quality reasoning paths for enhanced long-context performance. Our framework incorporates a self-sampling mechanism to bootstrap reasoning paths and a novel quality assessment protocol specifically designed for long-context scenarios. Experimental results on various long-context benchmarks demonstrate the effectiveness of our approach, achieving significant improvements over outcome supervision baselines on both in-domain tasks (+13.6/+3.8 points for LLaMA/Qwen on MuSiQue) and cross-domain generalization (+9.3/+8.1 points on average across diverse QA tasks). Our code, data and trained models are made public to facilitate future research. |
| title | Chain-of-Thought Matters: Improving Long-Context Language Models with Reasoning Path Supervision |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.20790 |