Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2602.03560 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910010460602368 |
|---|---|
| author | Gao, Yizhao Wei, Jianyu Zhang, Qihao Cheng, Yu Chen, Shimao Tang, Zhengju Jiang, Zihan Song, Yifan Zhang, Hailin Zhao, Liang Yang, Bo Wang, Gang Cao, Shijie Luo, Fuli |
| author_facet | Gao, Yizhao Wei, Jianyu Zhang, Qihao Cheng, Yu Chen, Shimao Tang, Zhengju Jiang, Zihan Song, Yifan Zhang, Hailin Zhao, Liang Yang, Bo Wang, Gang Cao, Shijie Luo, Fuli |
| contents | This work introduces Hybrid Sparse Attention (HySparse), a new architecture that interleaves each full attention layer with several sparse attention layers. While conceptually simple, HySparse strategically derives each sparse layer's token selection and KV caches directly from the preceding full attention layer. This architecture resolves two fundamental limitations of prior sparse attention methods. First, conventional approaches typically rely on additional proxies to predict token importance, introducing extra complexity and potentially suboptimal performance. In contrast, HySparse uses the full attention layer as a precise oracle to identify important tokens. Second, existing sparse attention designs often reduce computation without saving KV cache. HySparse enables sparse attention layers to reuse the full attention KV cache, thereby reducing both computation and memory. We evaluate HySparse on both 7B dense and 80B MoE models. Across all settings, HySparse consistently outperforms both full attention and hybrid SWA baselines. Notably, in the 80B MoE model with 49 total layers, only 5 layers employ full attention, yet HySparse achieves substantial performance gains while reducing KV cache storage by nearly 10x. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_03560 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing Gao, Yizhao Wei, Jianyu Zhang, Qihao Cheng, Yu Chen, Shimao Tang, Zhengju Jiang, Zihan Song, Yifan Zhang, Hailin Zhao, Liang Yang, Bo Wang, Gang Cao, Shijie Luo, Fuli Computation and Language Artificial Intelligence This work introduces Hybrid Sparse Attention (HySparse), a new architecture that interleaves each full attention layer with several sparse attention layers. While conceptually simple, HySparse strategically derives each sparse layer's token selection and KV caches directly from the preceding full attention layer. This architecture resolves two fundamental limitations of prior sparse attention methods. First, conventional approaches typically rely on additional proxies to predict token importance, introducing extra complexity and potentially suboptimal performance. In contrast, HySparse uses the full attention layer as a precise oracle to identify important tokens. Second, existing sparse attention designs often reduce computation without saving KV cache. HySparse enables sparse attention layers to reuse the full attention KV cache, thereby reducing both computation and memory. We evaluate HySparse on both 7B dense and 80B MoE models. Across all settings, HySparse consistently outperforms both full attention and hybrid SWA baselines. Notably, in the 80B MoE model with 49 total layers, only 5 layers employ full attention, yet HySparse achieves substantial performance gains while reducing KV cache storage by nearly 10x. |
| title | HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2602.03560 |