GQKVA: Efficient Pre-training of Transformers by Grouping Queries, Keys, and Values
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909621445197824 |
|---|---|
| author | Javadi, Farnoosh Ahmed, Walid Hajimolahoseini, Habib Ataiefard, Foozhan Hassanpour, Mohammad Asani, Saina Wen, Austin Awad, Omar Mohamed Liu, Kangling Liu, Yang |
| author_facet | Javadi, Farnoosh Ahmed, Walid Hajimolahoseini, Habib Ataiefard, Foozhan Hassanpour, Mohammad Asani, Saina Wen, Austin Awad, Omar Mohamed Liu, Kangling Liu, Yang |
| contents | Massive transformer-based models face several challenges, including slow and computationally intensive pre-training and over-parametrization. This paper addresses these challenges by proposing a versatile method called GQKVA, which generalizes query, key, and value grouping techniques. GQKVA is designed to speed up transformer pre-training while reducing the model size. Our experiments with various GQKVA variants highlight a clear trade-off between performance and model size, allowing for customized choices based on resource and time limitations. Our findings also indicate that the conventional multi-head attention approach is not always the best choice, as there are lighter and faster alternatives available. We tested our method on ViT, which achieved an approximate 0.3% increase in accuracy while reducing the model size by about 4% in the task of image classification. Additionally, our most aggressive model reduction experiment resulted in a reduction of approximately 15% in model size, with only around a 1% drop in accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_03426 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | GQKVA: Efficient Pre-training of Transformers by Grouping Queries, Keys, and Values Javadi, Farnoosh Ahmed, Walid Hajimolahoseini, Habib Ataiefard, Foozhan Hassanpour, Mohammad Asani, Saina Wen, Austin Awad, Omar Mohamed Liu, Kangling Liu, Yang Machine Learning Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Massive transformer-based models face several challenges, including slow and computationally intensive pre-training and over-parametrization. This paper addresses these challenges by proposing a versatile method called GQKVA, which generalizes query, key, and value grouping techniques. GQKVA is designed to speed up transformer pre-training while reducing the model size. Our experiments with various GQKVA variants highlight a clear trade-off between performance and model size, allowing for customized choices based on resource and time limitations. Our findings also indicate that the conventional multi-head attention approach is not always the best choice, as there are lighter and faster alternatives available. We tested our method on ViT, which achieved an approximate 0.3% increase in accuracy while reducing the model size by about 4% in the task of image classification. Additionally, our most aggressive model reduction experiment resulted in a reduction of approximately 15% in model size, with only around a 1% drop in accuracy. |
| title | GQKVA: Efficient Pre-training of Transformers by Grouping Queries, Keys, and Values |
| topic | Machine Learning Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2311.03426 |