ImPart: Importance-Aware Delta-Sparsification for Improved Model Compression and Merging in LLMs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917989430853632 |
|---|---|
| author | Yang, Yan Li, Yixia Wang, Hongru Wei, Xuetao Yu, Jianqiao Chen, Yun Chen, Guanhua |
| author_facet | Yang, Yan Li, Yixia Wang, Hongru Wei, Xuetao Yu, Jianqiao Chen, Yun Chen, Guanhua |
| contents | With the proliferation of task-specific large language models, delta compression has emerged as a method to mitigate the resource challenges of deploying numerous such models by effectively compressing the delta model parameters. Previous delta-sparsification methods either remove parameters randomly or truncate singular vectors directly after singular value decomposition (SVD). However, these methods either disregard parameter importance entirely or evaluate it with too coarse a granularity. In this work, we introduce ImPart, a novel importance-aware delta sparsification approach. Leveraging SVD, it dynamically adjusts sparsity ratios of different singular vectors based on their importance, effectively retaining crucial task-specific knowledge even at high sparsity ratios. Experiments show that ImPart achieves state-of-the-art delta sparsification performance, demonstrating $2\times$ higher compression ratio than baselines at the same performance level. When integrated with existing methods, ImPart sets a new state-of-the-art on delta quantization and model merging. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_13237 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ImPart: Importance-Aware Delta-Sparsification for Improved Model Compression and Merging in LLMs Yang, Yan Li, Yixia Wang, Hongru Wei, Xuetao Yu, Jianqiao Chen, Yun Chen, Guanhua Computation and Language With the proliferation of task-specific large language models, delta compression has emerged as a method to mitigate the resource challenges of deploying numerous such models by effectively compressing the delta model parameters. Previous delta-sparsification methods either remove parameters randomly or truncate singular vectors directly after singular value decomposition (SVD). However, these methods either disregard parameter importance entirely or evaluate it with too coarse a granularity. In this work, we introduce ImPart, a novel importance-aware delta sparsification approach. Leveraging SVD, it dynamically adjusts sparsity ratios of different singular vectors based on their importance, effectively retaining crucial task-specific knowledge even at high sparsity ratios. Experiments show that ImPart achieves state-of-the-art delta sparsification performance, demonstrating $2\times$ higher compression ratio than baselines at the same performance level. When integrated with existing methods, ImPart sets a new state-of-the-art on delta quantization and model merging. |
| title | ImPart: Importance-Aware Delta-Sparsification for Improved Model Compression and Merging in LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.13237 |