A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917234970984448 |
|---|---|
| author | Qiu, Zihan Huang, Zeyu Wen, Kaiyue Jin, Peng Zheng, Bo Zhou, Yuxin Huang, Haofeng Wang, Zekun Li, Xiao Zhang, Huaqing Xu, Yang Lian, Haoran Zhang, Siqi Men, Rui Zhang, Jianwei Titov, Ivan Liu, Dayiheng Zhou, Jingren Lin, Junyang |
| author_facet | Qiu, Zihan Huang, Zeyu Wen, Kaiyue Jin, Peng Zheng, Bo Zhou, Yuxin Huang, Haofeng Wang, Zekun Li, Xiao Zhang, Huaqing Xu, Yang Lian, Haoran Zhang, Siqi Men, Rui Zhang, Jianwei Titov, Ivan Liu, Dayiheng Zhou, Jingren Lin, Junyang |
| contents | We investigate the functional role of emergent outliers in large language models, specifically attention sinks (a few tokens that consistently receive large attention logits) and residual sinks (a few fixed dimensions with persistently large activations across most tokens). We hypothesize that these outliers, in conjunction with the corresponding normalizations (\textit{e.g.}, softmax attention and RMSNorm), effectively rescale other non-outlier components. We term this phenomenon \textit{outlier-driven rescaling} and validate this hypothesis across different model architectures and training token counts. This view unifies the origin and mitigation of both sink types. Our main conclusions and observations include: (1) Outliers function jointly with normalization: removing normalization eliminates the corresponding outliers but degrades training stability and performance; directly clipping outliers while retaining normalization leads to degradation, indicating that outlier-driven rescaling contributes to training stability. (2) Outliers serve more as rescale factors rather than contributors, as the final contributions of attention and residual sinks are significantly smaller than those of non-outliers. (3) Outliers can be absorbed into learnable parameters or mitigated via explicit gated rescaling, leading to improved training performance (average gain of 2 points) and enhanced quantization robustness (1.2 points degradation under W4A4 quantization). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_22966 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training Qiu, Zihan Huang, Zeyu Wen, Kaiyue Jin, Peng Zheng, Bo Zhou, Yuxin Huang, Haofeng Wang, Zekun Li, Xiao Zhang, Huaqing Xu, Yang Lian, Haoran Zhang, Siqi Men, Rui Zhang, Jianwei Titov, Ivan Liu, Dayiheng Zhou, Jingren Lin, Junyang Computation and Language We investigate the functional role of emergent outliers in large language models, specifically attention sinks (a few tokens that consistently receive large attention logits) and residual sinks (a few fixed dimensions with persistently large activations across most tokens). We hypothesize that these outliers, in conjunction with the corresponding normalizations (\textit{e.g.}, softmax attention and RMSNorm), effectively rescale other non-outlier components. We term this phenomenon \textit{outlier-driven rescaling} and validate this hypothesis across different model architectures and training token counts. This view unifies the origin and mitigation of both sink types. Our main conclusions and observations include: (1) Outliers function jointly with normalization: removing normalization eliminates the corresponding outliers but degrades training stability and performance; directly clipping outliers while retaining normalization leads to degradation, indicating that outlier-driven rescaling contributes to training stability. (2) Outliers serve more as rescale factors rather than contributors, as the final contributions of attention and residual sinks are significantly smaller than those of non-outliers. (3) Outliers can be absorbed into learnable parameters or mitigated via explicit gated rescaling, leading to improved training performance (average gain of 2 points) and enhanced quantization robustness (1.2 points degradation under W4A4 quantization). |
| title | A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.22966 |