Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefilling
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913162207428608 |
|---|---|
| author | Chen, Yujie Chen, Tailai Gao, Yifeng He, Zoe Wanying Xu, Yijue Wang, Shaobo Zhang, Linfeng |
| author_facet | Chen, Yujie Chen, Tailai Gao, Yifeng He, Zoe Wanying Xu, Yijue Wang, Shaobo Zhang, Linfeng |
| contents | Prefilling computational costs pose a significant bottleneck for Large Language Models (LLMs) and Large Multimodal Models (LMMs) in long-context settings. While token pruning reduces sequence length, prior methods rely on heuristics that break compatibility with hardware-efficient kernels like FlashAttention. In this work, we observe that tokens evolve toward \textit{semantic fixing points}, making further processing redundant. To this end, we introduce Delta Attention Selective Halting (DASH), a training-free policy that monitors the layer-wise update dynamics of the self-attention mechanism to selectively halt stabilized tokens. Extensive evaluation confirms that DASH generalizes across language and vision benchmarks, delivering significant prefill speedups while preserving model accuracy and hardware efficiency. Code will be released at https://github.com/verach3n/DASH.git. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_18103 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefilling Chen, Yujie Chen, Tailai Gao, Yifeng He, Zoe Wanying Xu, Yijue Wang, Shaobo Zhang, Linfeng Artificial Intelligence Prefilling computational costs pose a significant bottleneck for Large Language Models (LLMs) and Large Multimodal Models (LMMs) in long-context settings. While token pruning reduces sequence length, prior methods rely on heuristics that break compatibility with hardware-efficient kernels like FlashAttention. In this work, we observe that tokens evolve toward \textit{semantic fixing points}, making further processing redundant. To this end, we introduce Delta Attention Selective Halting (DASH), a training-free policy that monitors the layer-wise update dynamics of the self-attention mechanism to selectively halt stabilized tokens. Extensive evaluation confirms that DASH generalizes across language and vision benchmarks, delivering significant prefill speedups while preserving model accuracy and hardware efficiency. Code will be released at https://github.com/verach3n/DASH.git. |
| title | Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefilling |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2604.18103 |