Elastic Attention: Test-time Adaptive Sparsity Ratios for Efficient Transformers
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915758514110464 |
|---|---|
| author | Tang, Zecheng Qiu, Quantong Yang, Yi Hong, Zhiyi Xiang, Haiya Liu, Kebin Dang, Qingqing Li, Juntao Zhang, Min |
| author_facet | Tang, Zecheng Qiu, Quantong Yang, Yi Hong, Zhiyi Xiang, Haiya Liu, Kebin Dang, Qingqing Li, Juntao Zhang, Min |
| contents | The quadratic complexity of standard attention mechanisms poses a significant scalability bottleneck for large language models (LLMs) in long-context scenarios. While hybrid attention strategies that combine sparse and full attention within a single model offer a viable solution, they typically employ static computation ratios (i.e., fixed proportions of sparse versus full attention) and fail to adapt to the varying sparsity sensitivities of downstream tasks during inference. To address this issue, we propose Elastic Attention, which allows the model to dynamically adjust its overall sparsity based on the input. This is achieved by integrating a lightweight Attention Router into the existing pretrained model, which dynamically assigns each attention head to different computation modes. Within only 12 hours of training on 8xA800 GPUs, our method enables models to achieve both strong performance and efficient inference. Experiments across three long-context benchmarks on widely-used LLMs demonstrate the superiority of our method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_17367 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Elastic Attention: Test-time Adaptive Sparsity Ratios for Efficient Transformers Tang, Zecheng Qiu, Quantong Yang, Yi Hong, Zhiyi Xiang, Haiya Liu, Kebin Dang, Qingqing Li, Juntao Zhang, Min Computation and Language Artificial Intelligence The quadratic complexity of standard attention mechanisms poses a significant scalability bottleneck for large language models (LLMs) in long-context scenarios. While hybrid attention strategies that combine sparse and full attention within a single model offer a viable solution, they typically employ static computation ratios (i.e., fixed proportions of sparse versus full attention) and fail to adapt to the varying sparsity sensitivities of downstream tasks during inference. To address this issue, we propose Elastic Attention, which allows the model to dynamically adjust its overall sparsity based on the input. This is achieved by integrating a lightweight Attention Router into the existing pretrained model, which dynamically assigns each attention head to different computation modes. Within only 12 hours of training on 8xA800 GPUs, our method enables models to achieve both strong performance and efficient inference. Experiments across three long-context benchmarks on widely-used LLMs demonstrate the superiority of our method. |
| title | Elastic Attention: Test-time Adaptive Sparsity Ratios for Efficient Transformers |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2601.17367 |