Understanding the Difficulty of Low-Precision Post-Training Quantization for LLMs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908325285724160 |
|---|---|
| author | Xu, Zifei Sharify, Sayeh Yazar, Wanzin Webb, Tristan Wang, Xin |
| author_facet | Xu, Zifei Sharify, Sayeh Yazar, Wanzin Webb, Tristan Wang, Xin |
| contents | Large language models of high parameter counts are computationally expensive, yet can be made much more efficient by compressing their weights to very low numerical precision. This can be achieved either through post-training quantization by minimizing local, layer-wise quantization errors, or through quantization-aware fine-tuning by minimizing the global loss function. In this study, we discovered that, under the same data constraint, the former approach nearly always fared worse than the latter, a phenomenon particularly prominent when the numerical precision is very low. We further showed that this difficulty of post-training quantization arose from stark misalignment between optimization of the local and global objective functions. Our findings explains limited utility in minimization of local quantization error and the importance of direct quantization-aware fine-tuning, in the regime of large models at very low precision. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_14570 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Understanding the Difficulty of Low-Precision Post-Training Quantization for LLMs Xu, Zifei Sharify, Sayeh Yazar, Wanzin Webb, Tristan Wang, Xin Machine Learning Large language models of high parameter counts are computationally expensive, yet can be made much more efficient by compressing their weights to very low numerical precision. This can be achieved either through post-training quantization by minimizing local, layer-wise quantization errors, or through quantization-aware fine-tuning by minimizing the global loss function. In this study, we discovered that, under the same data constraint, the former approach nearly always fared worse than the latter, a phenomenon particularly prominent when the numerical precision is very low. We further showed that this difficulty of post-training quantization arose from stark misalignment between optimization of the local and global objective functions. Our findings explains limited utility in minimization of local quantization error and the importance of direct quantization-aware fine-tuning, in the regime of large models at very low precision. |
| title | Understanding the Difficulty of Low-Precision Post-Training Quantization for LLMs |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2410.14570 |