Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918022550126592 |
|---|---|
| author | Liu, Deyuan Qin, Zhanyue Wang, Hairu Yang, Zhao Wang, Zecheng Rong, Fangying Liu, Qingbin Hao, Yanchao Chen, Xi Fan, Cunhang Lv, Zhao Tu, Zhiying Chu, Dianhui Li, Bo Sui, Dianbo |
| author_facet | Liu, Deyuan Qin, Zhanyue Wang, Hairu Yang, Zhao Wang, Zecheng Rong, Fangying Liu, Qingbin Hao, Yanchao Chen, Xi Fan, Cunhang Lv, Zhao Tu, Zhiying Chu, Dianhui Li, Bo Sui, Dianbo |
| contents | While large language models (LLMs) excel in many domains, their complexity and scale challenge deployment in resource-limited environments. Current compression techniques, such as parameter pruning, often fail to effectively utilize the knowledge from pruned parameters. To address these challenges, we propose Manifold-Based Knowledge Alignment and Layer Merging Compression (MKA), a novel approach that uses manifold learning and the Normalized Pairwise Information Bottleneck (NPIB) measure to merge similar layers, reducing model size while preserving essential performance. We evaluate MKA on multiple benchmark datasets and various LLMs. Our findings show that MKA not only preserves model performance but also achieves substantial compression ratios, outperforming traditional pruning methods. Moreover, when coupled with quantization, MKA delivers even greater compression. Specifically, on the MMLU dataset using the Llama3-8B model, MKA achieves a compression ratio of 43.75% with a minimal performance decrease of only 2.82\%. The proposed MKA method offers a resource-efficient and performance-preserving model compression technique for LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_16330 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging Liu, Deyuan Qin, Zhanyue Wang, Hairu Yang, Zhao Wang, Zecheng Rong, Fangying Liu, Qingbin Hao, Yanchao Chen, Xi Fan, Cunhang Lv, Zhao Tu, Zhiying Chu, Dianhui Li, Bo Sui, Dianbo Computation and Language Artificial Intelligence While large language models (LLMs) excel in many domains, their complexity and scale challenge deployment in resource-limited environments. Current compression techniques, such as parameter pruning, often fail to effectively utilize the knowledge from pruned parameters. To address these challenges, we propose Manifold-Based Knowledge Alignment and Layer Merging Compression (MKA), a novel approach that uses manifold learning and the Normalized Pairwise Information Bottleneck (NPIB) measure to merge similar layers, reducing model size while preserving essential performance. We evaluate MKA on multiple benchmark datasets and various LLMs. Our findings show that MKA not only preserves model performance but also achieves substantial compression ratios, outperforming traditional pruning methods. Moreover, when coupled with quantization, MKA delivers even greater compression. Specifically, on the MMLU dataset using the Llama3-8B model, MKA achieves a compression ratio of 43.75% with a minimal performance decrease of only 2.82\%. The proposed MKA method offers a resource-efficient and performance-preserving model compression technique for LLMs. |
| title | Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2406.16330 |