DipSVD: Dual-importance Protected SVD for Efficient LLM Compression
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909659926888448 |
|---|---|
| author | Ding, Xuan Sun, Rui Zhang, Yunjian Yan, Xiu Zhou, Yueqi Huang, Kaihao Fu, Suzhong Xie, Chuanlong Zhu, Yao |
| author_facet | Ding, Xuan Sun, Rui Zhang, Yunjian Yan, Xiu Zhou, Yueqi Huang, Kaihao Fu, Suzhong Xie, Chuanlong Zhu, Yao |
| contents | The ever-increasing computational demands and deployment costs of large language models (LLMs) have spurred numerous compressing methods. Compared to quantization and unstructured pruning, SVD compression offers superior hardware compatibility and theoretical guarantees. However, existing SVD-based methods focus on the overall discrepancy between the original and compressed matrices while overlooking the protection of critical components within the matrix, which leads to inferior performance in the compressed models. This paper proposes a dual-level importance protection mechanism to enhance SVD-based compression methods: (1) local importance protection: preserving the most critical singular vectors within each weight matrix through channel-weighted data whitening; and (2) global importance protection: enabling less important layers to bear a greater portion of the compression burden through either a heuristic or optimization-based approach, thereby minimizing the impact of compression on critical layers. Extensive experiments demonstrate that DipSVD outperforms existing SVD-based compression approaches across multiple benchmarks, achieving superior model performance especially at high model compression ratios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_20353 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DipSVD: Dual-importance Protected SVD for Efficient LLM Compression Ding, Xuan Sun, Rui Zhang, Yunjian Yan, Xiu Zhou, Yueqi Huang, Kaihao Fu, Suzhong Xie, Chuanlong Zhu, Yao Machine Learning Artificial Intelligence The ever-increasing computational demands and deployment costs of large language models (LLMs) have spurred numerous compressing methods. Compared to quantization and unstructured pruning, SVD compression offers superior hardware compatibility and theoretical guarantees. However, existing SVD-based methods focus on the overall discrepancy between the original and compressed matrices while overlooking the protection of critical components within the matrix, which leads to inferior performance in the compressed models. This paper proposes a dual-level importance protection mechanism to enhance SVD-based compression methods: (1) local importance protection: preserving the most critical singular vectors within each weight matrix through channel-weighted data whitening; and (2) global importance protection: enabling less important layers to bear a greater portion of the compression burden through either a heuristic or optimization-based approach, thereby minimizing the impact of compression on critical layers. Extensive experiments demonstrate that DipSVD outperforms existing SVD-based compression approaches across multiple benchmarks, achieving superior model performance especially at high model compression ratios. |
| title | DipSVD: Dual-importance Protected SVD for Efficient LLM Compression |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2506.20353 |