Towards Next-Level Post-Training Quantization of Hyper-Scale Transformers
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915004886810624 |
|---|---|
| author | Kim, Junhan Lee, Chungman Cho, Eulrang Park, Kyungphil Kim, Ho-young Kim, Joonyoung Jeon, Yongkweon |
| author_facet | Kim, Junhan Lee, Chungman Cho, Eulrang Park, Kyungphil Kim, Ho-young Kim, Joonyoung Jeon, Yongkweon |
| contents | With the increasing complexity of generative AI models, post-training quantization (PTQ) has emerged as a promising solution for deploying hyper-scale models on edge devices such as mobile and TVs. Existing PTQ schemes, however, consume considerable time and resources, which could be a bottleneck in real situations where frequent model updates and multiple hyperparameter tunings are required. As a cost-effective alternative, learning-free PTQ schemes have been proposed. However, the performance is somewhat limited because they cannot consider the inter-layer dependency within the attention module, which is a significant feature of Transformers. In this paper, we thus propose a novel PTQ algorithm that balances accuracy and efficiency. The key idea of the proposed algorithm called aespa is to perform quantization layer-wise for efficiency while targeting attention-wise reconstruction to consider the cross-layer dependency. Through extensive experiments on various language models and complexity analysis, we demonstrate that aespa is accurate and efficient in quantizing Transformer models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_08958 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Next-Level Post-Training Quantization of Hyper-Scale Transformers Kim, Junhan Lee, Chungman Cho, Eulrang Park, Kyungphil Kim, Ho-young Kim, Joonyoung Jeon, Yongkweon Machine Learning Artificial Intelligence With the increasing complexity of generative AI models, post-training quantization (PTQ) has emerged as a promising solution for deploying hyper-scale models on edge devices such as mobile and TVs. Existing PTQ schemes, however, consume considerable time and resources, which could be a bottleneck in real situations where frequent model updates and multiple hyperparameter tunings are required. As a cost-effective alternative, learning-free PTQ schemes have been proposed. However, the performance is somewhat limited because they cannot consider the inter-layer dependency within the attention module, which is a significant feature of Transformers. In this paper, we thus propose a novel PTQ algorithm that balances accuracy and efficiency. The key idea of the proposed algorithm called aespa is to perform quantization layer-wise for efficiency while targeting attention-wise reconstruction to consider the cross-layer dependency. Through extensive experiments on various language models and complexity analysis, we demonstrate that aespa is accurate and efficient in quantizing Transformer models. |
| title | Towards Next-Level Post-Training Quantization of Hyper-Scale Transformers |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2402.08958 |