Attention Sinks and Outliers in Attention Residuals
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914577317363712 |
|---|---|
| author | Luo, Haozheng Dai, Haoran Zhang, Shaoyang Chen, Xi Jiang, Eric Hanchen Li, Yijiang Huang, Jingyuan Qiu, Chenghao Xu, Chenwei Pan, Zhenyu Zhang, Haotian Wang, Binghui Chen, Yan |
| author_facet | Luo, Haozheng Dai, Haoran Zhang, Shaoyang Chen, Xi Jiang, Eric Hanchen Li, Yijiang Huang, Jingyuan Qiu, Chenghao Xu, Chenwei Pan, Zhenyu Zhang, Haotian Wang, Binghui Chen, Yan |
| contents | We propose OASIS, an outlier- and sink-aware technique built on inter-layer null signaling. As AttnResidual architectures introduce an additional depth-wise normalization channel, they improve inter-layer routing flexibility but also exacerbate attention sinks, activation outliers, and the resulting degradation in inference stability and quantization robustness. OASIS addresses this issue by introducing a Softmax1-based null space and coupling token-level null evidence to depth routing through an inter-layer null signal, thereby reducing sink-dominated routing and improving structural robustness. Theoretically, we show that the dual-normalization design of AttnResidual intensifies sink formation and quantization brittleness. Experimentally, we compare OASIS against five baselines on three real-world datasets and observe consistent improvements in both attention sink and post-quantization performance. Notably, OASIS achieves an average reduction of 9.26% in maximum infinity norm and 2.60% in average kurtosis across the evaluated settings, while lowering perplexity by 75.85% under W8A8 and improving GSM8K Pass@1 by 12.42% under W4A4. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_17887 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Attention Sinks and Outliers in Attention Residuals Luo, Haozheng Dai, Haoran Zhang, Shaoyang Chen, Xi Jiang, Eric Hanchen Li, Yijiang Huang, Jingyuan Qiu, Chenghao Xu, Chenwei Pan, Zhenyu Zhang, Haotian Wang, Binghui Chen, Yan Machine Learning Artificial Intelligence We propose OASIS, an outlier- and sink-aware technique built on inter-layer null signaling. As AttnResidual architectures introduce an additional depth-wise normalization channel, they improve inter-layer routing flexibility but also exacerbate attention sinks, activation outliers, and the resulting degradation in inference stability and quantization robustness. OASIS addresses this issue by introducing a Softmax1-based null space and coupling token-level null evidence to depth routing through an inter-layer null signal, thereby reducing sink-dominated routing and improving structural robustness. Theoretically, we show that the dual-normalization design of AttnResidual intensifies sink formation and quantization brittleness. Experimentally, we compare OASIS against five baselines on three real-world datasets and observe consistent improvements in both attention sink and post-quantization performance. Notably, OASIS achieves an average reduction of 9.26% in maximum infinity norm and 2.60% in average kurtosis across the evaluated settings, while lowering perplexity by 75.85% under W8A8 and improving GSM8K Pass@1 by 12.42% under W4A4. |
| title | Attention Sinks and Outliers in Attention Residuals |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2605.17887 |