DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929572169121792 |
|---|---|
| author | Lin, Haokun Xu, Haobo Wu, Yichen Cui, Jingzhi Zhang, Yingtao Mou, Linzhan Song, Linqi Sun, Zhenan Wei, Ying |
| author_facet | Lin, Haokun Xu, Haobo Wu, Yichen Cui, Jingzhi Zhang, Yingtao Mou, Linzhan Song, Linqi Sun, Zhenan Wei, Ying |
| contents | Quantization of large language models (LLMs) faces significant challenges, particularly due to the presence of outlier activations that impede efficient low-bit representation. Traditional approaches predominantly address Normal Outliers, which are activations across all tokens with relatively large magnitudes. However, these methods struggle with smoothing Massive Outliers that display significantly larger values, which leads to significant performance degradation in low-bit quantization. In this paper, we introduce DuQuant, a novel approach that utilizes rotation and permutation transformations to more effectively mitigate both massive and normal outliers. First, DuQuant starts by constructing the rotation matrix, using specific outlier dimensions as prior knowledge, to redistribute outliers to adjacent channels by block-wise rotation. Second, We further employ a zigzag permutation to balance the distribution of outliers across blocks, thereby reducing block-wise variance. A subsequent rotation further smooths the activation landscape, enhancing model performance. DuQuant simplifies the quantization process and excels in managing outliers, outperforming the state-of-the-art baselines across various sizes and types of LLMs on multiple tasks, even with 4-bit weight-activation quantization. Our code is available at https://github.com/Hsu1023/DuQuant. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_01721 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs Lin, Haokun Xu, Haobo Wu, Yichen Cui, Jingzhi Zhang, Yingtao Mou, Linzhan Song, Linqi Sun, Zhenan Wei, Ying Computation and Language Quantization of large language models (LLMs) faces significant challenges, particularly due to the presence of outlier activations that impede efficient low-bit representation. Traditional approaches predominantly address Normal Outliers, which are activations across all tokens with relatively large magnitudes. However, these methods struggle with smoothing Massive Outliers that display significantly larger values, which leads to significant performance degradation in low-bit quantization. In this paper, we introduce DuQuant, a novel approach that utilizes rotation and permutation transformations to more effectively mitigate both massive and normal outliers. First, DuQuant starts by constructing the rotation matrix, using specific outlier dimensions as prior knowledge, to redistribute outliers to adjacent channels by block-wise rotation. Second, We further employ a zigzag permutation to balance the distribution of outliers across blocks, thereby reducing block-wise variance. A subsequent rotation further smooths the activation landscape, enhancing model performance. DuQuant simplifies the quantization process and excels in managing outliers, outperforming the state-of-the-art baselines across various sizes and types of LLMs on multiple tasks, even with 4-bit weight-activation quantization. Our code is available at https://github.com/Hsu1023/DuQuant. |
| title | DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.01721 |