LongAttn: Selecting Long-context Training Data via Token-level Attention
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913709714046976 |
|---|---|
| author | Wu, Longyun Zhu, Dawei Zhao, Guangxiang Yu, Zhuocheng Ran, Junfeng Wong, Xiangyu Sun, Lin Li, Sujian |
| author_facet | Wu, Longyun Zhu, Dawei Zhao, Guangxiang Yu, Zhuocheng Ran, Junfeng Wong, Xiangyu Sun, Lin Li, Sujian |
| contents | With the development of large language models (LLMs), there has been an increasing need for significant advancements in handling long contexts. To enhance long-context capabilities, constructing high-quality training data with long-range dependencies is crucial. Existing methods to select long-context data often rely on sentence-level analysis, which can be greatly optimized in both performance and efficiency. In this paper, we propose a novel token-level framework, LongAttn, which leverages the self-attention mechanism of LLMs to measure the long-range dependencies for the data. By calculating token-level dependency strength and distribution uniformity of token scores, LongAttn effectively quantifies long-range dependencies, enabling more accurate and efficient data selection. We filter LongABC-32K from open-source long-context datasets (ArXiv, Book, and Code). Through our comprehensive experiments, LongAttn has demonstrated its excellent effectiveness, scalability, and efficiency. To facilitate future research in long-context data, we released our code and the high-quality long-context training data LongABC-32K. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_16860 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LongAttn: Selecting Long-context Training Data via Token-level Attention Wu, Longyun Zhu, Dawei Zhao, Guangxiang Yu, Zhuocheng Ran, Junfeng Wong, Xiangyu Sun, Lin Li, Sujian Computation and Language With the development of large language models (LLMs), there has been an increasing need for significant advancements in handling long contexts. To enhance long-context capabilities, constructing high-quality training data with long-range dependencies is crucial. Existing methods to select long-context data often rely on sentence-level analysis, which can be greatly optimized in both performance and efficiency. In this paper, we propose a novel token-level framework, LongAttn, which leverages the self-attention mechanism of LLMs to measure the long-range dependencies for the data. By calculating token-level dependency strength and distribution uniformity of token scores, LongAttn effectively quantifies long-range dependencies, enabling more accurate and efficient data selection. We filter LongABC-32K from open-source long-context datasets (ArXiv, Book, and Code). Through our comprehensive experiments, LongAttn has demonstrated its excellent effectiveness, scalability, and efficiency. To facilitate future research in long-context data, we released our code and the high-quality long-context training data LongABC-32K. |
| title | LongAttn: Selecting Long-context Training Data via Token-level Attention |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.16860 |