LiteLong: Resource-Efficient Long-Context Data Synthesis for LLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915502651080704 |
|---|---|
| author | Jia, Junlong Wu, Xing Gao, Chaochen Chen, Ziyang Lin, Zijia Li, Zhongzhi Wang, Weinong Xu, Haotian Jin, Donghui Zhang, Debing Guo, Binghui |
| author_facet | Jia, Junlong Wu, Xing Gao, Chaochen Chen, Ziyang Lin, Zijia Li, Zhongzhi Wang, Weinong Xu, Haotian Jin, Donghui Zhang, Debing Guo, Binghui |
| contents | High-quality long-context data is essential for training large language models (LLMs) capable of processing extensive documents, yet existing synthesis approaches using relevance-based aggregation face challenges of computational efficiency. We present LiteLong, a resource-efficient method for synthesizing long-context data through structured topic organization and multi-agent debate. Our approach leverages the BISAC book classification system to provide a comprehensive hierarchical topic organization, and then employs a debate mechanism with multiple LLMs to generate diverse, high-quality topics within this structure. For each topic, we use lightweight BM25 retrieval to obtain relevant documents and concatenate them into 128K-token training samples. Experiments on HELMET and Ruler benchmarks demonstrate that LiteLong achieves competitive long-context performance and can seamlessly integrate with other long-dependency enhancement methods. LiteLong makes high-quality long-context data synthesis more accessible by reducing both computational and data engineering costs, facilitating further research in long-context language training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_15568 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LiteLong: Resource-Efficient Long-Context Data Synthesis for LLMs Jia, Junlong Wu, Xing Gao, Chaochen Chen, Ziyang Lin, Zijia Li, Zhongzhi Wang, Weinong Xu, Haotian Jin, Donghui Zhang, Debing Guo, Binghui Computation and Language Artificial Intelligence High-quality long-context data is essential for training large language models (LLMs) capable of processing extensive documents, yet existing synthesis approaches using relevance-based aggregation face challenges of computational efficiency. We present LiteLong, a resource-efficient method for synthesizing long-context data through structured topic organization and multi-agent debate. Our approach leverages the BISAC book classification system to provide a comprehensive hierarchical topic organization, and then employs a debate mechanism with multiple LLMs to generate diverse, high-quality topics within this structure. For each topic, we use lightweight BM25 retrieval to obtain relevant documents and concatenate them into 128K-token training samples. Experiments on HELMET and Ruler benchmarks demonstrate that LiteLong achieves competitive long-context performance and can seamlessly integrate with other long-dependency enhancement methods. LiteLong makes high-quality long-context data synthesis more accessible by reducing both computational and data engineering costs, facilitating further research in long-context language training. |
| title | LiteLong: Resource-Efficient Long-Context Data Synthesis for LLMs |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2509.15568 |