Round Attention: A Novel Round-Level Attention Mechanism to Accelerate LLM Inference
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916811960745984 |
|---|---|
| author | Tang, Yaohua Hu, Zhicheng Cheng, Kun Mo, Fan Lv, Qiheng Wang, Hua Chen, Zhi |
| author_facet | Tang, Yaohua Hu, Zhicheng Cheng, Kun Mo, Fan Lv, Qiheng Wang, Hua Chen, Zhi |
| contents | The increasing context window size in large language models (LLMs) has improved their ability to handle complex, long-text tasks. However, as the conversation rounds continue, it is required to store a large amount of KV cache in GPU memory, which significantly affects the efficiency and even availability of the model serving systems. This paper analyzes dialogue data from real users on the granularity of round and discovers that the LLM inference manifests a watershed layer, after which the distribution of round-level attention shows notable similarity. Based on this, we propose Round Attention - a novel round-level attention mechanism that selectively processes the KV cache of top-k relevant rounds, where k is dynamically determined through the attention matrix in the watershed layer. Theoretical analysis demonstrates that our method reduces memory usage by 54\% to 82\%, while experimental results confirm that loading sparse critical-round KV cache maintains answer accuracy without performance degradation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_15294 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Round Attention: A Novel Round-Level Attention Mechanism to Accelerate LLM Inference Tang, Yaohua Hu, Zhicheng Cheng, Kun Mo, Fan Lv, Qiheng Wang, Hua Chen, Zhi Computation and Language Artificial Intelligence The increasing context window size in large language models (LLMs) has improved their ability to handle complex, long-text tasks. However, as the conversation rounds continue, it is required to store a large amount of KV cache in GPU memory, which significantly affects the efficiency and even availability of the model serving systems. This paper analyzes dialogue data from real users on the granularity of round and discovers that the LLM inference manifests a watershed layer, after which the distribution of round-level attention shows notable similarity. Based on this, we propose Round Attention - a novel round-level attention mechanism that selectively processes the KV cache of top-k relevant rounds, where k is dynamically determined through the attention matrix in the watershed layer. Theoretical analysis demonstrates that our method reduces memory usage by 54\% to 82\%, while experimental results confirm that loading sparse critical-round KV cache maintains answer accuracy without performance degradation. |
| title | Round Attention: A Novel Round-Level Attention Mechanism to Accelerate LLM Inference |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.15294 |