Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912928444186624 |
|---|---|
| author | Bae, Jeongin Park, Baeseong Park, Gunho Kim, Minsub Lee, Joonhyung Yoo, Junhee Woo, Sunghyeon Ryu, Jiwon Kwon, Se Jung Lee, Dongsoo |
| author_facet | Bae, Jeongin Park, Baeseong Park, Gunho Kim, Minsub Lee, Joonhyung Yoo, Junhee Woo, Sunghyeon Ryu, Jiwon Kwon, Se Jung Lee, Dongsoo |
| contents | Transformer attention is typically implemented using softmax normalization, which enforces attention weights with unit sum normalization. While effective in many settings, this constraint can limit flexibility in controlling attention magnitudes and may contribute to overly concentrated or unstable attention patterns during training. Prior work has explored modifications such as attention sinks or gating mechanisms, but these approaches provide only limited or indirect control over attention reweighting. We propose Affine-Scaled Attention, a simple extension to standard attention that introduces input-dependent scaling and a corresponding bias term applied to softmax-normalized attention weights. This design relaxes the strict normalization constraint while maintaining aggregation of value representations, allowing the model to adjust both the relative distribution and the scale of attention in a controlled manner.
We empirically evaluate Affine-Scaled Attention in large-scale language model pretraining across multiple model sizes. Experimental results show consistent improvements in training stability, optimization behavior, and downstream task performance compared to standard softmax attention and attention sink baselines. These findings suggest that modest reweighting of attention outputs provides a practical and effective way to improve attention behavior in Transformer models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_23057 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention Bae, Jeongin Park, Baeseong Park, Gunho Kim, Minsub Lee, Joonhyung Yoo, Junhee Woo, Sunghyeon Ryu, Jiwon Kwon, Se Jung Lee, Dongsoo Computation and Language Artificial Intelligence Transformer attention is typically implemented using softmax normalization, which enforces attention weights with unit sum normalization. While effective in many settings, this constraint can limit flexibility in controlling attention magnitudes and may contribute to overly concentrated or unstable attention patterns during training. Prior work has explored modifications such as attention sinks or gating mechanisms, but these approaches provide only limited or indirect control over attention reweighting. We propose Affine-Scaled Attention, a simple extension to standard attention that introduces input-dependent scaling and a corresponding bias term applied to softmax-normalized attention weights. This design relaxes the strict normalization constraint while maintaining aggregation of value representations, allowing the model to adjust both the relative distribution and the scale of attention in a controlled manner. We empirically evaluate Affine-Scaled Attention in large-scale language model pretraining across multiple model sizes. Experimental results show consistent improvements in training stability, optimization behavior, and downstream task performance compared to standard softmax attention and attention sink baselines. These findings suggest that modest reweighting of attention outputs provides a practical and effective way to improve attention behavior in Transformer models. |
| title | Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2602.23057 |