Pause-Tuning for Long-Context Comprehension: A Lightweight Approach to LLM Attention Recalibration
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916633946095616 |
|---|---|
| author | Begin, James Agrawal, Namit Singh, Eshan Fu, Yicheng O'Brien, Sean Sharma, Vasu Zhu, Kevin |
| author_facet | Begin, James Agrawal, Namit Singh, Eshan Fu, Yicheng O'Brien, Sean Sharma, Vasu Zhu, Kevin |
| contents | LLMs have demonstrated remarkable proficiency in understanding tasks but continue to struggle with long-context comprehension, particularly with content located in the middle of extensive inputs. This limitation, known as the Lost-in-the-Middle (LITM) problem, hinders models from fully processing and utilizing information across lengthy contexts. To address this issue, we introduce pause-tuning, a technique that redistributes attention to enhance comprehension of long-context inputs. Our approach involves fine-tuning language models on datasets with artificially inserted pause tokens, which serve to segment the input into smaller, more manageable parts. We evaluate pause-tuning against alternative approaches using the Needle-in-a-Haystack benchmark, where models must retrieve information embedded within contexts of up to 128K tokens. Experimental results demonstrate significant performance gains, with the LLaMA 3.2 3B Instruct model and the LLaMA 3.1 8B Instruct model improving by 10.61% and 3.57% respectively on average, suggesting that pause-tuning successfully enhances attention redistribution and improves long-context retention. The code and data are available at https://anonymous.4open.science/r/LITM-PauseTokens-7357. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_20405 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Pause-Tuning for Long-Context Comprehension: A Lightweight Approach to LLM Attention Recalibration Begin, James Agrawal, Namit Singh, Eshan Fu, Yicheng O'Brien, Sean Sharma, Vasu Zhu, Kevin Computation and Language Artificial Intelligence LLMs have demonstrated remarkable proficiency in understanding tasks but continue to struggle with long-context comprehension, particularly with content located in the middle of extensive inputs. This limitation, known as the Lost-in-the-Middle (LITM) problem, hinders models from fully processing and utilizing information across lengthy contexts. To address this issue, we introduce pause-tuning, a technique that redistributes attention to enhance comprehension of long-context inputs. Our approach involves fine-tuning language models on datasets with artificially inserted pause tokens, which serve to segment the input into smaller, more manageable parts. We evaluate pause-tuning against alternative approaches using the Needle-in-a-Haystack benchmark, where models must retrieve information embedded within contexts of up to 128K tokens. Experimental results demonstrate significant performance gains, with the LLaMA 3.2 3B Instruct model and the LLaMA 3.1 8B Instruct model improving by 10.61% and 3.57% respectively on average, suggesting that pause-tuning successfully enhances attention redistribution and improves long-context retention. The code and data are available at https://anonymous.4open.science/r/LITM-PauseTokens-7357. |
| title | Pause-Tuning for Long-Context Comprehension: A Lightweight Approach to LLM Attention Recalibration |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.20405 |