RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915086296154112 |
|---|---|
| author | Liu, Di Chen, Meng Lu, Baotong Jiang, Huiqiang Han, Zhenhua Zhang, Qianxi Chen, Qi Zhang, Chengruidong Ding, Bailu Zhang, Kai Chen, Chen Yang, Fan Yang, Yuqing Qiu, Lili |
| author_facet | Liu, Di Chen, Meng Lu, Baotong Jiang, Huiqiang Han, Zhenhua Zhang, Qianxi Chen, Qi Zhang, Chengruidong Ding, Bailu Zhang, Kai Chen, Chen Yang, Fan Yang, Yuqing Qiu, Lili |
| contents | Transformer-based Large Language Models (LLMs) have become increasingly important. However, due to the quadratic time complexity of attention computation, scaling LLMs to longer contexts incurs extremely slow inference speed and high GPU memory consumption for caching key-value (KV) vectors. This paper proposes RetrievalAttention, a training-free approach to both accelerate attention computation and reduce GPU memory consumption. By leveraging the dynamic sparsity of attention mechanism, RetrievalAttention proposes to build approximate nearest neighbor search (ANNS) indexes for KV vectors in CPU memory and retrieve the most relevant ones through vector search during generation. Unfortunately, we observe that the off-the-shelf ANNS indexes are often ineffective for such retrieval tasks due to the out-of-distribution (OOD) between query vectors and key vectors in the attention mechanism. RetrievalAttention addresses the OOD challenge by designing an attention-aware vector search algorithm that can adapt to the distribution of query vectors. Our evaluation demonstrates that RetrievalAttention achieves near full attention accuracy while only requiring access to 1--3% of the data. This leads to a significant reduction in the inference cost of long-context LLMs, with a much lower GPU memory footprint. In particular, RetrievalAttention only needs a single NVIDIA RTX4090 (24GB) to serve 128K tokens for LLMs with 8B parameters, which is capable of generating one token in 0.188 seconds. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_10516 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval Liu, Di Chen, Meng Lu, Baotong Jiang, Huiqiang Han, Zhenhua Zhang, Qianxi Chen, Qi Zhang, Chengruidong Ding, Bailu Zhang, Kai Chen, Chen Yang, Fan Yang, Yuqing Qiu, Lili Machine Learning Computation and Language Transformer-based Large Language Models (LLMs) have become increasingly important. However, due to the quadratic time complexity of attention computation, scaling LLMs to longer contexts incurs extremely slow inference speed and high GPU memory consumption for caching key-value (KV) vectors. This paper proposes RetrievalAttention, a training-free approach to both accelerate attention computation and reduce GPU memory consumption. By leveraging the dynamic sparsity of attention mechanism, RetrievalAttention proposes to build approximate nearest neighbor search (ANNS) indexes for KV vectors in CPU memory and retrieve the most relevant ones through vector search during generation. Unfortunately, we observe that the off-the-shelf ANNS indexes are often ineffective for such retrieval tasks due to the out-of-distribution (OOD) between query vectors and key vectors in the attention mechanism. RetrievalAttention addresses the OOD challenge by designing an attention-aware vector search algorithm that can adapt to the distribution of query vectors. Our evaluation demonstrates that RetrievalAttention achieves near full attention accuracy while only requiring access to 1--3% of the data. This leads to a significant reduction in the inference cost of long-context LLMs, with a much lower GPU memory footprint. In particular, RetrievalAttention only needs a single NVIDIA RTX4090 (24GB) to serve 128K tokens for LLMs with 8B parameters, which is capable of generating one token in 0.188 seconds. |
| title | RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2409.10516 |