AWP: Activation-Aware Weight Pruning and Quantization with Projected Gradient Descent
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912737202798592 |
|---|---|
| author | Liu, Jing Koike-Akino, Toshiaki Wang, Ye Mansour, Hassan Brand, Matthew |
| author_facet | Liu, Jing Koike-Akino, Toshiaki Wang, Ye Mansour, Hassan Brand, Matthew |
| contents | To address the enormous size of Large Language Models (LLMs), model compression methods, such as quantization and pruning, are often deployed, especially on edge devices. In this work, we focus on layer-wise post-training quantization and pruning. Drawing connections between activation-aware weight pruning and sparse approximation problems, and motivated by the success of Iterative Hard Thresholding (IHT), we propose a unified method for Activation-aware Weight pruning and quantization via Projected gradient descent (AWP). Our experiments demonstrate that AWP outperforms state-of-the-art LLM pruning and quantization methods. Theoretical convergence guarantees of the proposed method for pruning are also provided. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_10205 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AWP: Activation-Aware Weight Pruning and Quantization with Projected Gradient Descent Liu, Jing Koike-Akino, Toshiaki Wang, Ye Mansour, Hassan Brand, Matthew Machine Learning To address the enormous size of Large Language Models (LLMs), model compression methods, such as quantization and pruning, are often deployed, especially on edge devices. In this work, we focus on layer-wise post-training quantization and pruning. Drawing connections between activation-aware weight pruning and sparse approximation problems, and motivated by the success of Iterative Hard Thresholding (IHT), we propose a unified method for Activation-aware Weight pruning and quantization via Projected gradient descent (AWP). Our experiments demonstrate that AWP outperforms state-of-the-art LLM pruning and quantization methods. Theoretical convergence guarantees of the proposed method for pruning are also provided. |
| title | AWP: Activation-Aware Weight Pruning and Quantization with Projected Gradient Descent |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2506.10205 |