A Survey on Transformer Context Extension: Approaches and Evaluation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913931431247872 |
|---|---|
| author | Liu, Yijun Yu, Jinzheng Xu, Yang Li, Zhongyang Zhu, Qingfu |
| author_facet | Liu, Yijun Yu, Jinzheng Xu, Yang Li, Zhongyang Zhu, Qingfu |
| contents | Large language models (LLMs) based on Transformer have been widely applied in the filed of natural language processing (NLP), demonstrating strong performance, particularly in handling short text tasks. However, when it comes to long context scenarios, the performance of LLMs degrades due to some challenges. To alleviate this phenomenon, there is a number of work proposed recently. In this survey, we first list the challenges of applying pre-trained LLMs to process long contexts. Then systematically review the approaches related to long context and propose our taxonomy categorizing them into four main types: positional encoding, context compression, retrieval augmented, and attention pattern. In addition to the approaches, we focus on the evaluation of long context, organizing relevant data, tasks, and metrics based on existing long context benchmarks. Finally, we summarize unresolved issues in the long context domain and put forward our views on future developments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_13299 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Survey on Transformer Context Extension: Approaches and Evaluation Liu, Yijun Yu, Jinzheng Xu, Yang Li, Zhongyang Zhu, Qingfu Computation and Language Artificial Intelligence Large language models (LLMs) based on Transformer have been widely applied in the filed of natural language processing (NLP), demonstrating strong performance, particularly in handling short text tasks. However, when it comes to long context scenarios, the performance of LLMs degrades due to some challenges. To alleviate this phenomenon, there is a number of work proposed recently. In this survey, we first list the challenges of applying pre-trained LLMs to process long contexts. Then systematically review the approaches related to long context and propose our taxonomy categorizing them into four main types: positional encoding, context compression, retrieval augmented, and attention pattern. In addition to the approaches, we focus on the evaluation of long context, organizing relevant data, tasks, and metrics based on existing long context benchmarks. Finally, we summarize unresolved issues in the long context domain and put forward our views on future developments. |
| title | A Survey on Transformer Context Extension: Approaches and Evaluation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2503.13299 |