Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2502.13963 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912238454964224 |
|---|---|
| author | Liu, Weihao Wu, Ning Yang, Shiping Ding, Wenbiao Liang, Shining Gong, Ming Zhang, Dongmei |
| author_facet | Liu, Weihao Wu, Ning Yang, Shiping Ding, Wenbiao Liang, Shining Gong, Ming Zhang, Dongmei |
| contents | Large Language Models (LLMs) frequently show distracted attention due to irrelevant information in the input, which severely impairs their long-context capabilities. Inspired by recent studies on the effectiveness of retrieval heads in long-context factutality, we aim at addressing this distraction issue through improving such retrieval heads directly. We propose Multi-Document Attention Focusing (MuDAF), a novel method that explicitly optimizes the attention distribution at the head level through contrastive learning. According to the experimental results, MuDAF can significantly improve the long-context question answering performance of LLMs, especially in multi-document question answering. Extensive evaluations on retrieval scores and attention visualizations show that MuDAF possesses great potential in making attention heads more focused on relevant information and reducing attention distractions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_13963 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MuDAF: Long-Context Multi-Document Attention Focusing through Contrastive Learning on Attention Heads Liu, Weihao Wu, Ning Yang, Shiping Ding, Wenbiao Liang, Shining Gong, Ming Zhang, Dongmei Computation and Language Large Language Models (LLMs) frequently show distracted attention due to irrelevant information in the input, which severely impairs their long-context capabilities. Inspired by recent studies on the effectiveness of retrieval heads in long-context factutality, we aim at addressing this distraction issue through improving such retrieval heads directly. We propose Multi-Document Attention Focusing (MuDAF), a novel method that explicitly optimizes the attention distribution at the head level through contrastive learning. According to the experimental results, MuDAF can significantly improve the long-context question answering performance of LLMs, especially in multi-document question answering. Extensive evaluations on retrieval scores and attention visualizations show that MuDAF possesses great potential in making attention heads more focused on relevant information and reducing attention distractions. |
| title | MuDAF: Long-Context Multi-Document Attention Focusing through Contrastive Learning on Attention Heads |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.13963 |