AttentionRetriever: Attention Layers are Secretly Long Document Retrievers
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911443857702912 |
|---|---|
| author | Fu, David Jiahao Do, Lam Thanh Li, Jiayu Chang, Kevin Chen-Chuan |
| author_facet | Fu, David Jiahao Do, Lam Thanh Li, Jiayu Chang, Kevin Chen-Chuan |
| contents | Retrieval augmented generation (RAG) has been widely adopted to help Large Language Models (LLMs) to process tasks involving long documents. However, existing retrieval models are not designed for long document retrieval and fail to address several key challenges of long document retrieval, including context-awareness, causal dependence, and scope of retrieval. In this paper, we proposed AttentionRetriever, a novel long document retrieval model that leverages attention mechanism and entity-based retrieval to build context-aware embeddings for long document and determine the scope of retrieval. With extensive experiments, we found AttentionRetriever is able to outperform existing retrieval models on long document retrieval datasets by a large margin while remaining as efficient as dense retrieval models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_12278 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | AttentionRetriever: Attention Layers are Secretly Long Document Retrievers Fu, David Jiahao Do, Lam Thanh Li, Jiayu Chang, Kevin Chen-Chuan Information Retrieval Artificial Intelligence Retrieval augmented generation (RAG) has been widely adopted to help Large Language Models (LLMs) to process tasks involving long documents. However, existing retrieval models are not designed for long document retrieval and fail to address several key challenges of long document retrieval, including context-awareness, causal dependence, and scope of retrieval. In this paper, we proposed AttentionRetriever, a novel long document retrieval model that leverages attention mechanism and entity-based retrieval to build context-aware embeddings for long document and determine the scope of retrieval. With extensive experiments, we found AttentionRetriever is able to outperform existing retrieval models on long document retrieval datasets by a large margin while remaining as efficient as dense retrieval models. |
| title | AttentionRetriever: Attention Layers are Secretly Long Document Retrievers |
| topic | Information Retrieval Artificial Intelligence |
| url | https://arxiv.org/abs/2602.12278 |