CBQ: Cross-Block Quantization for Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913706195025920 |
|---|---|
| author | Ding, Xin Liu, Xiaoyu Tu, Zhijun Zhang, Yun Li, Wei Hu, Jie Chen, Hanting Tang, Yehui Xiong, Zhiwei Yin, Baoqun Wang, Yunhe |
| author_facet | Ding, Xin Liu, Xiaoyu Tu, Zhijun Zhang, Yun Li, Wei Hu, Jie Chen, Hanting Tang, Yehui Xiong, Zhiwei Yin, Baoqun Wang, Yunhe |
| contents | Post-training quantization (PTQ) has played a key role in compressing large language models (LLMs) with ultra-low costs. However, existing PTQ methods only focus on handling the outliers within one layer or one block, which ignores the dependency of blocks and leads to severe performance degradation in low-bit settings. In this paper, we propose CBQ, a cross-block reconstruction-based PTQ method for LLMs. CBQ employs a cross-block dependency using a homologous reconstruction scheme, establishing long-range dependencies across multiple blocks to minimize error accumulation. Furthermore, CBQ incorporates a coarse-to-fine preprocessing (CFP) strategy for suppressing weight and activation outliers, coupled with an adaptive LoRA-Rounding technique for precise weight quantization. These innovations enable CBQ to not only handle extreme outliers effectively but also improve overall quantization accuracy. Extensive experiments show that CBQ achieves superior low-bit quantization (W4A4, W4A8, W2A16) and outperforms existing state-of-the-art methods across various LLMs and datasets. Notably, CBQ quantizes the 4-bit LLAMA1-65B model within only 4.3 hours on a single GPU, achieving a commendable tradeoff between performance and quantization efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_07950 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | CBQ: Cross-Block Quantization for Large Language Models Ding, Xin Liu, Xiaoyu Tu, Zhijun Zhang, Yun Li, Wei Hu, Jie Chen, Hanting Tang, Yehui Xiong, Zhiwei Yin, Baoqun Wang, Yunhe Machine Learning Computation and Language Post-training quantization (PTQ) has played a key role in compressing large language models (LLMs) with ultra-low costs. However, existing PTQ methods only focus on handling the outliers within one layer or one block, which ignores the dependency of blocks and leads to severe performance degradation in low-bit settings. In this paper, we propose CBQ, a cross-block reconstruction-based PTQ method for LLMs. CBQ employs a cross-block dependency using a homologous reconstruction scheme, establishing long-range dependencies across multiple blocks to minimize error accumulation. Furthermore, CBQ incorporates a coarse-to-fine preprocessing (CFP) strategy for suppressing weight and activation outliers, coupled with an adaptive LoRA-Rounding technique for precise weight quantization. These innovations enable CBQ to not only handle extreme outliers effectively but also improve overall quantization accuracy. Extensive experiments show that CBQ achieves superior low-bit quantization (W4A4, W4A8, W2A16) and outperforms existing state-of-the-art methods across various LLMs and datasets. Notably, CBQ quantizes the 4-bit LLAMA1-65B model within only 4.3 hours on a single GPU, achieving a commendable tradeoff between performance and quantization efficiency. |
| title | CBQ: Cross-Block Quantization for Large Language Models |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2312.07950 |