Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.10098 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911585166950400 |
|---|---|
| author | Su, Zunhai Zhang, Hengyuan Wu, Wei Zhang, Yifan Liu, Yaxiu Xiao, He Yang, Qingyao Sun, Yuxuan Yang, Rui Zhang, Chao Fan, Keyu Ye, Weihao Xiong, Jing Shen, Hui Tao, Chaofan Wu, Taiqiang Wan, Zhongwei Qian, Yulei Xie, Yuchen Wong, Ngai |
| author_facet | Su, Zunhai Zhang, Hengyuan Wu, Wei Zhang, Yifan Liu, Yaxiu Xiao, He Yang, Qingyao Sun, Yuxuan Yang, Rui Zhang, Chao Fan, Keyu Ye, Weihao Xiong, Jing Shen, Hui Tao, Chaofan Wu, Taiqiang Wan, Zhongwei Qian, Yulei Xie, Yuchen Wong, Ngai |
| contents | As the foundational architecture of modern machine learning, Transformers have driven remarkable progress across diverse AI domains. Despite their transformative impact, a persistent challenge across various Transformers is Attention Sink (AS), in which a disproportionate amount of attention is focused on a small subset of specific yet uninformative tokens. AS complicates interpretability, significantly affecting the training and inference dynamics, and exacerbates issues such as hallucinations. In recent years, substantial research has been dedicated to understanding and harnessing AS. However, a comprehensive survey that systematically consolidates AS-related research and offers guidance for future advancements remains lacking. To address this gap, we present the first survey on AS, structured around three key dimensions that define the current research landscape: Fundamental Utilization, Mechanistic Interpretation, and Strategic Mitigation. Our work provides a pivotal contribution by clarifying key concepts and guiding researchers through the evolution and trends of the field. We envision this survey as a definitive resource, empowering researchers and practitioners to effectively manage AS within the current Transformer paradigm, while simultaneously inspiring innovative advancements for the next generation of Transformers. The paper list of this work is available at https://github.com/ZunhaiSu/Awesome-Attention-Sink. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_10098 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation Su, Zunhai Zhang, Hengyuan Wu, Wei Zhang, Yifan Liu, Yaxiu Xiao, He Yang, Qingyao Sun, Yuxuan Yang, Rui Zhang, Chao Fan, Keyu Ye, Weihao Xiong, Jing Shen, Hui Tao, Chaofan Wu, Taiqiang Wan, Zhongwei Qian, Yulei Xie, Yuchen Wong, Ngai Machine Learning As the foundational architecture of modern machine learning, Transformers have driven remarkable progress across diverse AI domains. Despite their transformative impact, a persistent challenge across various Transformers is Attention Sink (AS), in which a disproportionate amount of attention is focused on a small subset of specific yet uninformative tokens. AS complicates interpretability, significantly affecting the training and inference dynamics, and exacerbates issues such as hallucinations. In recent years, substantial research has been dedicated to understanding and harnessing AS. However, a comprehensive survey that systematically consolidates AS-related research and offers guidance for future advancements remains lacking. To address this gap, we present the first survey on AS, structured around three key dimensions that define the current research landscape: Fundamental Utilization, Mechanistic Interpretation, and Strategic Mitigation. Our work provides a pivotal contribution by clarifying key concepts and guiding researchers through the evolution and trends of the field. We envision this survey as a definitive resource, empowering researchers and practitioners to effectively manage AS within the current Transformer paradigm, while simultaneously inspiring innovative advancements for the next generation of Transformers. The paper list of this work is available at https://github.com/ZunhaiSu/Awesome-Attention-Sink. |
| title | Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2604.10098 |