AttnCache: Accelerating Self-Attention Inference for LLM Prefill via Attention Cache
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915612627828736 |
|---|---|
| author | Song, Dinghong Feng, Yuan Wang, Yiwei Chen, Shangye Guyot, Cyril Blagojevic, Filip Jeon, Hyeran Su, Pengfei Li, Dong |
| author_facet | Song, Dinghong Feng, Yuan Wang, Yiwei Chen, Shangye Guyot, Cyril Blagojevic, Filip Jeon, Hyeran Su, Pengfei Li, Dong |
| contents | Large Language Models (LLMs) are widely used in generative applications such as chatting, code generation, and reasoning. However, many realworld workloads such as classification, question answering, recommendation, and text embedding rely solely on the prefill stage of inference, where the model encodes input sequences without performing autoregressive decoding. In these prefill only scenarios, the self-attention computation becomes the primary performance bottleneck due to its quadratic complexity with respect to sequence length. In this paper, we observe that semantically different sentences often produce similar attention maps across layers and heads. Building on this insight, we propose AttnCache, a framework that accelerates the prefill stage of LLM inference by retrieving and reusing similar attention maps. Based on an attention map memorization database, AttnCache employs efficient caching and similarity search techniques to identify and reuse pre-cached attention maps during inference, thereby reducing the computational overhead of self-attention. Experimental results show that AttnCache achieves an average of 1.2x end-to-end and 2x attention speedup on CPU, and 1.6x end-to-end and 3x attention speedup on GPU, with negligible accuracy degradation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_25979 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AttnCache: Accelerating Self-Attention Inference for LLM Prefill via Attention Cache Song, Dinghong Feng, Yuan Wang, Yiwei Chen, Shangye Guyot, Cyril Blagojevic, Filip Jeon, Hyeran Su, Pengfei Li, Dong Computation and Language Machine Learning Large Language Models (LLMs) are widely used in generative applications such as chatting, code generation, and reasoning. However, many realworld workloads such as classification, question answering, recommendation, and text embedding rely solely on the prefill stage of inference, where the model encodes input sequences without performing autoregressive decoding. In these prefill only scenarios, the self-attention computation becomes the primary performance bottleneck due to its quadratic complexity with respect to sequence length. In this paper, we observe that semantically different sentences often produce similar attention maps across layers and heads. Building on this insight, we propose AttnCache, a framework that accelerates the prefill stage of LLM inference by retrieving and reusing similar attention maps. Based on an attention map memorization database, AttnCache employs efficient caching and similarity search techniques to identify and reuse pre-cached attention maps during inference, thereby reducing the computational overhead of self-attention. Experimental results show that AttnCache achieves an average of 1.2x end-to-end and 2x attention speedup on CPU, and 1.6x end-to-end and 3x attention speedup on GPU, with negligible accuracy degradation. |
| title | AttnCache: Accelerating Self-Attention Inference for LLM Prefill via Attention Cache |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2510.25979 |