LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918159525609472 |
|---|---|
| author | Chen, Jianghao Wu, Junhong Xu, Yangyifan Zhang, Jiajun |
| author_facet | Chen, Jianghao Wu, Junhong Xu, Yangyifan Zhang, Jiajun |
| contents | Long-context modeling has drawn more and more attention in the area of Large Language Models (LLMs). Continual training with long-context data becomes the de-facto method to equip LLMs with the ability to process long inputs. However, it still remains an open challenge to measure the quality of long-context training data. To address this issue, we propose a Long-context data selection framework with Attention-based Dependency Measurement (LADM), which can efficiently identify high-quality long-context data from a large-scale, multi-domain pre-training corpus. LADM leverages the retrieval capabilities of the attention mechanism to capture contextual dependencies, ensuring a comprehensive quality measurement of long-context data. Experimental results show that our LADM framework significantly boosts the performance of LLMs on multiple long-context tasks with only 1B tokens for continual training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_02502 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs Chen, Jianghao Wu, Junhong Xu, Yangyifan Zhang, Jiajun Computation and Language Long-context modeling has drawn more and more attention in the area of Large Language Models (LLMs). Continual training with long-context data becomes the de-facto method to equip LLMs with the ability to process long inputs. However, it still remains an open challenge to measure the quality of long-context training data. To address this issue, we propose a Long-context data selection framework with Attention-based Dependency Measurement (LADM), which can efficiently identify high-quality long-context data from a large-scale, multi-domain pre-training corpus. LADM leverages the retrieval capabilities of the attention mechanism to capture contextual dependencies, ensuring a comprehensive quality measurement of long-context data. Experimental results show that our LADM framework significantly boosts the performance of LLMs on multiple long-context tasks with only 1B tokens for continual training. |
| title | LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2503.02502 |