Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2506.02572 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910982911033344 |
|---|---|
| author | Gong, Ping Yi, Jiawei Wang, Shengnan Zhang, Juncheng Jin, Zewen Zhou, Ouxiang Liu, Ruibo Xu, Guanbin Bai, Youhui Ye, Bowen Yuan, Kun Yang, Tong Zhang, Gong Chen, Renhai Wu, Feng Li, Cheng |
| author_facet | Gong, Ping Yi, Jiawei Wang, Shengnan Zhang, Juncheng Jin, Zewen Zhou, Ouxiang Liu, Ruibo Xu, Guanbin Bai, Youhui Ye, Bowen Yuan, Kun Yang, Tong Zhang, Gong Chen, Renhai Wu, Feng Li, Cheng |
| contents | Large Language Models (LLMs) have emerged as a pivotal research area, yet the attention module remains a critical bottleneck in LLM inference, even with techniques like KVCache to mitigate redundant computations. While various top-$k$ attention mechanisms have been proposed to accelerate LLM inference by exploiting the inherent sparsity of attention, they often struggled to strike a balance between efficiency and accuracy. In this paper, we introduce HATA (Hash-Aware Top-$k$ Attention), a novel approach that systematically integrates low-overhead learning-to-hash techniques into the Top-$k$ attention process. Different from the existing top-k attention methods which are devoted to seeking an absolute estimation of qk score, typically with a great cost, HATA maps queries and keys into binary hash codes, and acquires the relative qk score order with a quite low cost, which is sufficient for realizing top-k attention. Extensive experiments demonstrate that HATA achieves up to 7.2$\times$ speedup compared to vanilla full attention while maintaining model accuracy. In addition, HATA outperforms the state-of-the-art top-$k$ attention methods in both accuracy and efficiency across multiple mainstream LLM models and diverse tasks. HATA is open source at https://github.com/gpzlx1/HATA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_02572 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention for Scalable Large Model Inference Gong, Ping Yi, Jiawei Wang, Shengnan Zhang, Juncheng Jin, Zewen Zhou, Ouxiang Liu, Ruibo Xu, Guanbin Bai, Youhui Ye, Bowen Yuan, Kun Yang, Tong Zhang, Gong Chen, Renhai Wu, Feng Li, Cheng Machine Learning Artificial Intelligence Large Language Models (LLMs) have emerged as a pivotal research area, yet the attention module remains a critical bottleneck in LLM inference, even with techniques like KVCache to mitigate redundant computations. While various top-$k$ attention mechanisms have been proposed to accelerate LLM inference by exploiting the inherent sparsity of attention, they often struggled to strike a balance between efficiency and accuracy. In this paper, we introduce HATA (Hash-Aware Top-$k$ Attention), a novel approach that systematically integrates low-overhead learning-to-hash techniques into the Top-$k$ attention process. Different from the existing top-k attention methods which are devoted to seeking an absolute estimation of qk score, typically with a great cost, HATA maps queries and keys into binary hash codes, and acquires the relative qk score order with a quite low cost, which is sufficient for realizing top-k attention. Extensive experiments demonstrate that HATA achieves up to 7.2$\times$ speedup compared to vanilla full attention while maintaining model accuracy. In addition, HATA outperforms the state-of-the-art top-$k$ attention methods in both accuracy and efficiency across multiple mainstream LLM models and diverse tasks. HATA is open source at https://github.com/gpzlx1/HATA. |
| title | HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention for Scalable Large Model Inference |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2506.02572 |