GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912597979168768 |
|---|---|
| author | Kim, Jinuk Halabi, Marwa El Park, Wonpyo Schaefer, Clemens JS Lee, Deokjae Park, Yeonhong Lee, Jae W. Song, Hyun Oh |
| author_facet | Kim, Jinuk Halabi, Marwa El Park, Wonpyo Schaefer, Clemens JS Lee, Deokjae Park, Yeonhong Lee, Jae W. Song, Hyun Oh |
| contents | Post-training quantization is a key technique for reducing the memory and inference latency of large language models by quantizing weights and activations without requiring retraining. However, existing methods either (1) fail to account for the varying importance of hidden features to the end loss or, when incorporating end loss, (2) neglect the critical interactions between model weights. To address these limitations, we propose GuidedQuant, a novel quantization approach that integrates gradient information from the end loss into the quantization objective while preserving cross-weight dependencies within output channels. GuidedQuant consistently boosts the performance of state-of-the-art quantization methods across weight-only scalar, weight-only vector, and weight-and-activation quantization. Additionally, we introduce a novel non-uniform scalar quantization algorithm, which is guaranteed to monotonically decrease the quantization objective value, and outperforms existing methods in this category. We release the code at https://github.com/snu-mllab/GuidedQuant. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_07004 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance Kim, Jinuk Halabi, Marwa El Park, Wonpyo Schaefer, Clemens JS Lee, Deokjae Park, Yeonhong Lee, Jae W. Song, Hyun Oh Machine Learning Post-training quantization is a key technique for reducing the memory and inference latency of large language models by quantizing weights and activations without requiring retraining. However, existing methods either (1) fail to account for the varying importance of hidden features to the end loss or, when incorporating end loss, (2) neglect the critical interactions between model weights. To address these limitations, we propose GuidedQuant, a novel quantization approach that integrates gradient information from the end loss into the quantization objective while preserving cross-weight dependencies within output channels. GuidedQuant consistently boosts the performance of state-of-the-art quantization methods across weight-only scalar, weight-only vector, and weight-and-activation quantization. Additionally, we introduce a novel non-uniform scalar quantization algorithm, which is guaranteed to monotonically decrease the quantization objective value, and outperforms existing methods in this category. We release the code at https://github.com/snu-mllab/GuidedQuant. |
| title | GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2505.07004 |