A Comprehensive Survey on Long Context Language Modeling
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908672352845824 |
|---|---|
| author | Liu, Jiaheng Zhu, Dawei Bai, Zhiqi He, Yancheng Liao, Huanxuan Que, Haoran Wang, Zekun Zhang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, Yuanxing Chen, Zhuo Guo, Hangyu Li, Shilong Liu, Ziqiang Shan, Yong Song, Yifan Tian, Jiayi Wu, Wenhao Zhou, Zhejian Zhu, Ruijie Feng, Junlan Gao, Yang He, Shizhu Li, Zhoujun Liu, Tianyu Meng, Fanyu Su, Wenbo Tan, Yingshui Wang, Zili Yang, Jian Ye, Wei Zheng, Bo Zhou, Wangchunshu Huang, Wenhao Li, Sujian Zhang, Zhaoxiang |
| author_facet | Liu, Jiaheng Zhu, Dawei Bai, Zhiqi He, Yancheng Liao, Huanxuan Que, Haoran Wang, Zekun Zhang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, Yuanxing Chen, Zhuo Guo, Hangyu Li, Shilong Liu, Ziqiang Shan, Yong Song, Yifan Tian, Jiayi Wu, Wenhao Zhou, Zhejian Zhu, Ruijie Feng, Junlan Gao, Yang He, Shizhu Li, Zhoujun Liu, Tianyu Meng, Fanyu Su, Wenbo Tan, Yingshui Wang, Zili Yang, Jian Ye, Wei Zheng, Bo Zhou, Wangchunshu Huang, Wenhao Li, Sujian Zhang, Zhaoxiang |
| contents | Efficient processing of long contexts has been a persistent pursuit in Natural Language Processing. With the growing number of long documents, dialogues, and other textual data, it is important to develop Long Context Language Models (LCLMs) that can process and analyze extensive inputs in an effective and efficient way. In this paper, we present a comprehensive survey on recent advances in long-context modeling for large language models. Our survey is structured around three key aspects: how to obtain effective and efficient LCLMs, how to train and deploy LCLMs efficiently, and how to evaluate and analyze LCLMs comprehensively. For the first aspect, we discuss data strategies, architectural designs, and workflow approaches oriented with long context processing. For the second aspect, we provide a detailed examination of the infrastructure required for LCLM training and inference. For the third aspect, we present evaluation paradigms for long-context comprehension and long-form generation, as well as behavioral analysis and mechanism interpretability of LCLMs. Beyond these three key aspects, we thoroughly explore the diverse application scenarios where existing LCLMs have been deployed and outline promising future development directions. This survey provides an up-to-date review of the literature on long-context LLMs, which we wish to serve as a valuable resource for both researchers and engineers. An associated GitHub repository collecting the latest papers and repos is available at: \href{https://github.com/LCLM-Horizon/A-Comprehensive-Survey-For-Long-Context-Language-Modeling}{\color[RGB]{175,36,67}{LCLM-Horizon}}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_17407 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Comprehensive Survey on Long Context Language Modeling Liu, Jiaheng Zhu, Dawei Bai, Zhiqi He, Yancheng Liao, Huanxuan Que, Haoran Wang, Zekun Zhang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, Yuanxing Chen, Zhuo Guo, Hangyu Li, Shilong Liu, Ziqiang Shan, Yong Song, Yifan Tian, Jiayi Wu, Wenhao Zhou, Zhejian Zhu, Ruijie Feng, Junlan Gao, Yang He, Shizhu Li, Zhoujun Liu, Tianyu Meng, Fanyu Su, Wenbo Tan, Yingshui Wang, Zili Yang, Jian Ye, Wei Zheng, Bo Zhou, Wangchunshu Huang, Wenhao Li, Sujian Zhang, Zhaoxiang Computation and Language Machine Learning Efficient processing of long contexts has been a persistent pursuit in Natural Language Processing. With the growing number of long documents, dialogues, and other textual data, it is important to develop Long Context Language Models (LCLMs) that can process and analyze extensive inputs in an effective and efficient way. In this paper, we present a comprehensive survey on recent advances in long-context modeling for large language models. Our survey is structured around three key aspects: how to obtain effective and efficient LCLMs, how to train and deploy LCLMs efficiently, and how to evaluate and analyze LCLMs comprehensively. For the first aspect, we discuss data strategies, architectural designs, and workflow approaches oriented with long context processing. For the second aspect, we provide a detailed examination of the infrastructure required for LCLM training and inference. For the third aspect, we present evaluation paradigms for long-context comprehension and long-form generation, as well as behavioral analysis and mechanism interpretability of LCLMs. Beyond these three key aspects, we thoroughly explore the diverse application scenarios where existing LCLMs have been deployed and outline promising future development directions. This survey provides an up-to-date review of the literature on long-context LLMs, which we wish to serve as a valuable resource for both researchers and engineers. An associated GitHub repository collecting the latest papers and repos is available at: \href{https://github.com/LCLM-Horizon/A-Comprehensive-Survey-For-Long-Context-Language-Modeling}{\color[RGB]{175,36,67}{LCLM-Horizon}}. |
| title | A Comprehensive Survey on Long Context Language Modeling |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2503.17407 |