Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.14963 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916296396898304 |
|---|---|
| author | Chen, Yuang Zhang, Cheng Gao, Xitong Mullins, Robert D. Constantinides, George A. Zhao, Yiren |
| author_facet | Chen, Yuang Zhang, Cheng Gao, Xitong Mullins, Robert D. Constantinides, George A. Zhao, Yiren |
| contents | Grouped-query attention (GQA) has been widely adopted in LLMs to mitigate the complexity of multi-head attention (MHA). To transform an MHA to a GQA, neighbour queries in MHA are evenly split into groups where each group shares the value and key layers. In this work, we propose AsymGQA, an activation-informed approach to asymmetrically grouping an MHA to a GQA for better model performance. Our AsymGQA outperforms the GQA within the same model size budget. For example, AsymGQA LLaMA-2-7B has an accuracy increase of 7.5% on MMLU compared to neighbour grouping. Our approach addresses the GQA's trade-off problem between model performance and hardware efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_14963 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Optimised Grouped-Query Attention Mechanism for Transformers Chen, Yuang Zhang, Cheng Gao, Xitong Mullins, Robert D. Constantinides, George A. Zhao, Yiren Machine Learning Grouped-query attention (GQA) has been widely adopted in LLMs to mitigate the complexity of multi-head attention (MHA). To transform an MHA to a GQA, neighbour queries in MHA are evenly split into groups where each group shares the value and key layers. In this work, we propose AsymGQA, an activation-informed approach to asymmetrically grouping an MHA to a GQA for better model performance. Our AsymGQA outperforms the GQA within the same model size budget. For example, AsymGQA LLaMA-2-7B has an accuracy increase of 7.5% on MMLU compared to neighbour grouping. Our approach addresses the GQA's trade-off problem between model performance and hardware efficiency. |
| title | Optimised Grouped-Query Attention Mechanism for Transformers |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2406.14963 |