Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2402.10631 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916128277659648 |
|---|---|
| author | Du, Dayou Zhang, Yijia Cao, Shijie Guo, Jiaqi Cao, Ting Chu, Xiaowen Xu, Ningyi |
| author_facet | Du, Dayou Zhang, Yijia Cao, Shijie Guo, Jiaqi Cao, Ting Chu, Xiaowen Xu, Ningyi |
| contents | The upscaling of Large Language Models (LLMs) has yielded impressive advances in natural language processing, yet it also poses significant deployment challenges. Weight quantization has emerged as a widely embraced solution to reduce memory and computational demands. This paper introduces BitDistiller, a framework that synergizes Quantization-Aware Training (QAT) with Knowledge Distillation (KD) to boost the performance of LLMs at ultra-low precisions (sub-4-bit). Specifically, BitDistiller first incorporates a tailored asymmetric quantization and clipping technique to maximally preserve the fidelity of quantized weights, and then proposes a novel Confidence-Aware Kullback-Leibler Divergence (CAKLD) objective, which is employed in a self-distillation manner to enable faster convergence and superior model performance. Empirical evaluations demonstrate that BitDistiller significantly surpasses existing methods in both 3-bit and 2-bit configurations on general language understanding and complex reasoning benchmarks. Notably, BitDistiller is shown to be more cost-effective, demanding fewer data and training resources. The code is available at https://github.com/DD-DuDa/BitDistiller. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_10631 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation Du, Dayou Zhang, Yijia Cao, Shijie Guo, Jiaqi Cao, Ting Chu, Xiaowen Xu, Ningyi Computation and Language The upscaling of Large Language Models (LLMs) has yielded impressive advances in natural language processing, yet it also poses significant deployment challenges. Weight quantization has emerged as a widely embraced solution to reduce memory and computational demands. This paper introduces BitDistiller, a framework that synergizes Quantization-Aware Training (QAT) with Knowledge Distillation (KD) to boost the performance of LLMs at ultra-low precisions (sub-4-bit). Specifically, BitDistiller first incorporates a tailored asymmetric quantization and clipping technique to maximally preserve the fidelity of quantized weights, and then proposes a novel Confidence-Aware Kullback-Leibler Divergence (CAKLD) objective, which is employed in a self-distillation manner to enable faster convergence and superior model performance. Empirical evaluations demonstrate that BitDistiller significantly surpasses existing methods in both 3-bit and 2-bit configurations on general language understanding and complex reasoning benchmarks. Notably, BitDistiller is shown to be more cost-effective, demanding fewer data and training resources. The code is available at https://github.com/DD-DuDa/BitDistiller. |
| title | BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.10631 |