Integer Scale: A Free Lunch for Faster Fine-grained Quantization of LLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929361809047552 |
|---|---|
| author | Li, Qingyuan Meng, Ran Li, Yiduo Zhang, Bo Lu, Yifan Sun, Yerui Ma, Lin Xie, Yuchen |
| author_facet | Li, Qingyuan Meng, Ran Li, Yiduo Zhang, Bo Lu, Yifan Sun, Yerui Ma, Lin Xie, Yuchen |
| contents | We introduce Integer Scale, a novel post-training quantization scheme for large language models that effectively resolves the inference bottleneck in current fine-grained quantization approaches while maintaining similar accuracies. Integer Scale is a free lunch as it requires no extra calibration or fine-tuning which will otherwise incur additional costs. It can be used plug-and-play for most fine-grained quantization methods. Its integration results in at most 1.85x end-to-end speed boost over the original counterpart with comparable accuracy. Additionally, due to the orchestration of the proposed Integer Scale and fine-grained quantization, we resolved the quantization difficulty for Mixtral-8x7B and LLaMA-3 models with negligible performance degradation, and it comes with an end-to-end speed boost of 2.13x, and 2.31x compared with their FP16 versions respectively. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_14597 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Integer Scale: A Free Lunch for Faster Fine-grained Quantization of LLMs Li, Qingyuan Meng, Ran Li, Yiduo Zhang, Bo Lu, Yifan Sun, Yerui Ma, Lin Xie, Yuchen Machine Learning Artificial Intelligence We introduce Integer Scale, a novel post-training quantization scheme for large language models that effectively resolves the inference bottleneck in current fine-grained quantization approaches while maintaining similar accuracies. Integer Scale is a free lunch as it requires no extra calibration or fine-tuning which will otherwise incur additional costs. It can be used plug-and-play for most fine-grained quantization methods. Its integration results in at most 1.85x end-to-end speed boost over the original counterpart with comparable accuracy. Additionally, due to the orchestration of the proposed Integer Scale and fine-grained quantization, we resolved the quantization difficulty for Mixtral-8x7B and LLaMA-3 models with negligible performance degradation, and it comes with an end-to-end speed boost of 2.13x, and 2.31x compared with their FP16 versions respectively. |
| title | Integer Scale: A Free Lunch for Faster Fine-grained Quantization of LLMs |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2405.14597 |