FlatQuant: Flatness Matters for LLM Quantization
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913981448323072 |
|---|---|
| author | Sun, Yuxuan Liu, Ruikang Bai, Haoli Bao, Han Zhao, Kang Li, Yuening Hu, Jiaxin Yu, Xianzhi Hou, Lu Yuan, Chun Jiang, Xin Liu, Wulong Yao, Jun |
| author_facet | Sun, Yuxuan Liu, Ruikang Bai, Haoli Bao, Han Zhao, Kang Li, Yuening Hu, Jiaxin Yu, Xianzhi Hou, Lu Yuan, Chun Jiang, Xin Liu, Wulong Yao, Jun |
| contents | Recently, quantization has been widely used for the compression and acceleration of large language models (LLMs). Due to the outliers in LLMs, it is crucial to flatten weights and activations to minimize quantization error with equally spaced quantization points. Prior research explores various pre-quantization transformations to suppress outliers, such as per-channel scaling and Hadamard transformation. However, we observe that these transformed weights and activations can still exhibit steep and dispersed distributions. In this paper, we propose FlatQuant (Fast and Learnable Affine Transformation), a new post-training quantization approach that enhances the flatness of weights and activations. Our approach identifies optimal affine transformations for each linear layer, calibrated in hours via a lightweight objective. To reduce runtime overhead of affine transformation, we apply Kronecker product with two lightweight matrices, and fuse all operations in FlatQuant into a single kernel. Extensive experiments demonstrate that FlatQuant establishes a new state-of-the-art benchmark for quantization. For example, it achieves less than 1\% accuracy drop for W4A4 quantization on the LLaMA-3-70B model, surpassing SpinQuant by 7.5\%. Additionally, it provides up to 2.3x prefill speedup and 1.7x decoding speedup compared to the FP16 model. Code is available at: https://github.com/ruikangliu/FlatQuant. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_09426 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FlatQuant: Flatness Matters for LLM Quantization Sun, Yuxuan Liu, Ruikang Bai, Haoli Bao, Han Zhao, Kang Li, Yuening Hu, Jiaxin Yu, Xianzhi Hou, Lu Yuan, Chun Jiang, Xin Liu, Wulong Yao, Jun Computation and Language Machine Learning Recently, quantization has been widely used for the compression and acceleration of large language models (LLMs). Due to the outliers in LLMs, it is crucial to flatten weights and activations to minimize quantization error with equally spaced quantization points. Prior research explores various pre-quantization transformations to suppress outliers, such as per-channel scaling and Hadamard transformation. However, we observe that these transformed weights and activations can still exhibit steep and dispersed distributions. In this paper, we propose FlatQuant (Fast and Learnable Affine Transformation), a new post-training quantization approach that enhances the flatness of weights and activations. Our approach identifies optimal affine transformations for each linear layer, calibrated in hours via a lightweight objective. To reduce runtime overhead of affine transformation, we apply Kronecker product with two lightweight matrices, and fuse all operations in FlatQuant into a single kernel. Extensive experiments demonstrate that FlatQuant establishes a new state-of-the-art benchmark for quantization. For example, it achieves less than 1\% accuracy drop for W4A4 quantization on the LLaMA-3-70B model, surpassing SpinQuant by 7.5\%. Additionally, it provides up to 2.3x prefill speedup and 1.7x decoding speedup compared to the FP16 model. Code is available at: https://github.com/ruikangliu/FlatQuant. |
| title | FlatQuant: Flatness Matters for LLM Quantization |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2410.09426 |