QwenLong-CPRS: Towards $\infty$-LLMs with Dynamic Context Optimization
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912397343588352 |
|---|---|
| author | Shen, Weizhou Li, Chenliang Wan, Fanqi Liao, Shengyi Lai, Shaopeng Zhang, Bo Shi, Yingcheng Wu, Yuning Fu, Gang Li, Zhansheng Yang, Bin Zhang, Ji Huang, Fei Zhou, Jingren Yan, Ming |
| author_facet | Shen, Weizhou Li, Chenliang Wan, Fanqi Liao, Shengyi Lai, Shaopeng Zhang, Bo Shi, Yingcheng Wu, Yuning Fu, Gang Li, Zhansheng Yang, Bin Zhang, Ji Huang, Fei Zhou, Jingren Yan, Ming |
| contents | This technical report presents QwenLong-CPRS, a context compression framework designed for explicit long-context optimization, addressing prohibitive computation overhead during the prefill stage and the "lost in the middle" performance degradation of large language models (LLMs) during long sequence processing. Implemented through a novel dynamic context optimization mechanism, QwenLong-CPRS enables multi-granularity context compression guided by natural language instructions, achieving both efficiency gains and improved performance.
Evolved from the Qwen architecture series, QwenLong-CPRS introduces four key innovations: (1) Natural language-guided dynamic optimization, (2) Bidirectional reasoning layers for enhanced boundary awareness, (3) Token critic mechanisms with language modeling heads, and (4) Window-parallel inference.
Comprehensive evaluations across five benchmarks (4K-2M word contexts) demonstrate QwenLong-CPRS's threefold effectiveness: (1) Consistent superiority over other context management methods like RAG and sparse attention in both accuracy and efficiency. (2) Architecture-agnostic integration with all flagship LLMs, including GPT-4o, Gemini2.0-pro, Claude3.7-sonnet, DeepSeek-v3, and Qwen2.5-max, achieves 21.59$\times$ context compression alongside 19.15-point average performance gains; (3) Deployed with Qwen2.5-32B-Instruct, QwenLong-CPRS surpasses leading proprietary LLMs by 4.85 and 10.88 points on Ruler-128K and InfiniteBench, establishing new SOTA performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_18092 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | QwenLong-CPRS: Towards $\infty$-LLMs with Dynamic Context Optimization Shen, Weizhou Li, Chenliang Wan, Fanqi Liao, Shengyi Lai, Shaopeng Zhang, Bo Shi, Yingcheng Wu, Yuning Fu, Gang Li, Zhansheng Yang, Bin Zhang, Ji Huang, Fei Zhou, Jingren Yan, Ming Computation and Language This technical report presents QwenLong-CPRS, a context compression framework designed for explicit long-context optimization, addressing prohibitive computation overhead during the prefill stage and the "lost in the middle" performance degradation of large language models (LLMs) during long sequence processing. Implemented through a novel dynamic context optimization mechanism, QwenLong-CPRS enables multi-granularity context compression guided by natural language instructions, achieving both efficiency gains and improved performance. Evolved from the Qwen architecture series, QwenLong-CPRS introduces four key innovations: (1) Natural language-guided dynamic optimization, (2) Bidirectional reasoning layers for enhanced boundary awareness, (3) Token critic mechanisms with language modeling heads, and (4) Window-parallel inference. Comprehensive evaluations across five benchmarks (4K-2M word contexts) demonstrate QwenLong-CPRS's threefold effectiveness: (1) Consistent superiority over other context management methods like RAG and sparse attention in both accuracy and efficiency. (2) Architecture-agnostic integration with all flagship LLMs, including GPT-4o, Gemini2.0-pro, Claude3.7-sonnet, DeepSeek-v3, and Qwen2.5-max, achieves 21.59$\times$ context compression alongside 19.15-point average performance gains; (3) Deployed with Qwen2.5-32B-Instruct, QwenLong-CPRS surpasses leading proprietary LLMs by 4.85 and 10.88 points on Ruler-128K and InfiniteBench, establishing new SOTA performance. |
| title | QwenLong-CPRS: Towards $\infty$-LLMs with Dynamic Context Optimization |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.18092 |