SAS: Simulated Attention Score
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914169658277888 |
|---|---|
| author | Zheng, Chuanyang Sun, Jiankai Gao, Yihang Wang, Yuehao Wang, Peihao Xiong, Jing Ren, Liliang Cheng, Hao Kulkarni, Janardhan Shen, Yelong Wang, Atlas Schwager, Mac Schneider, Anderson Liu, Xiaodong Gao, Jianfeng |
| author_facet | Zheng, Chuanyang Sun, Jiankai Gao, Yihang Wang, Yuehao Wang, Peihao Xiong, Jing Ren, Liliang Cheng, Hao Kulkarni, Janardhan Shen, Yelong Wang, Atlas Schwager, Mac Schneider, Anderson Liu, Xiaodong Gao, Jianfeng |
| contents | The attention mechanism is a core component of the Transformer architecture. Various methods have been developed to compute attention scores, including multi-head attention (MHA), multi-query attention, group-query attention and so on. We further analyze the MHA and observe that its performance improves as the number of attention heads increases, provided the hidden size per head remains sufficiently large. Therefore, increasing both the head count and hidden size per head with minimal parameter overhead can lead to significant performance gains at a low cost. Motivated by this insight, we introduce Simulated Attention Score (SAS), which maintains a compact model size while simulating a larger number of attention heads and hidden feature dimension per head. This is achieved by projecting a low-dimensional head representation into a higher-dimensional space, effectively increasing attention capacity without increasing parameter count. Beyond the head representations, we further extend the simulation approach to feature dimension of the key and query embeddings, enhancing expressiveness by mimicking the behavior of a larger model while preserving the original model size. To control the parameter cost, we also propose Parameter-Efficient Attention Aggregation (PEAA). Comprehensive experiments on a variety of datasets and tasks demonstrate the effectiveness of the proposed SAS method, achieving significant improvements over different attention variants. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_07694 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SAS: Simulated Attention Score Zheng, Chuanyang Sun, Jiankai Gao, Yihang Wang, Yuehao Wang, Peihao Xiong, Jing Ren, Liliang Cheng, Hao Kulkarni, Janardhan Shen, Yelong Wang, Atlas Schwager, Mac Schneider, Anderson Liu, Xiaodong Gao, Jianfeng Computation and Language The attention mechanism is a core component of the Transformer architecture. Various methods have been developed to compute attention scores, including multi-head attention (MHA), multi-query attention, group-query attention and so on. We further analyze the MHA and observe that its performance improves as the number of attention heads increases, provided the hidden size per head remains sufficiently large. Therefore, increasing both the head count and hidden size per head with minimal parameter overhead can lead to significant performance gains at a low cost. Motivated by this insight, we introduce Simulated Attention Score (SAS), which maintains a compact model size while simulating a larger number of attention heads and hidden feature dimension per head. This is achieved by projecting a low-dimensional head representation into a higher-dimensional space, effectively increasing attention capacity without increasing parameter count. Beyond the head representations, we further extend the simulation approach to feature dimension of the key and query embeddings, enhancing expressiveness by mimicking the behavior of a larger model while preserving the original model size. To control the parameter cost, we also propose Parameter-Efficient Attention Aggregation (PEAA). Comprehensive experiments on a variety of datasets and tasks demonstrate the effectiveness of the proposed SAS method, achieving significant improvements over different attention variants. |
| title | SAS: Simulated Attention Score |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2507.07694 |