Forget the Data and Fine-Tuning! Just Fold the Network to Compress
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913985181253632 |
|---|---|
| author | Wang, Dong Šikić, Haris Thiele, Lothar Saukh, Olga |
| author_facet | Wang, Dong Šikić, Haris Thiele, Lothar Saukh, Olga |
| contents | We introduce model folding, a novel data-free model compression technique that merges structurally similar neurons across layers, significantly reducing the model size without the need for fine-tuning or access to training data. Unlike existing methods, model folding preserves data statistics during compression by leveraging k-means clustering, and using novel data-free techniques to prevent variance collapse or explosion. Our theoretical framework and experiments across standard benchmarks, including ResNet18 and LLaMA-7B, demonstrate that model folding achieves comparable performance to data-driven compression techniques and outperforms recently proposed data-free methods, especially at high sparsity levels. This approach is particularly effective for compressing large-scale models, making it suitable for deployment in resource-constrained environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_10216 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Forget the Data and Fine-Tuning! Just Fold the Network to Compress Wang, Dong Šikić, Haris Thiele, Lothar Saukh, Olga Machine Learning Artificial Intelligence We introduce model folding, a novel data-free model compression technique that merges structurally similar neurons across layers, significantly reducing the model size without the need for fine-tuning or access to training data. Unlike existing methods, model folding preserves data statistics during compression by leveraging k-means clustering, and using novel data-free techniques to prevent variance collapse or explosion. Our theoretical framework and experiments across standard benchmarks, including ResNet18 and LLaMA-7B, demonstrate that model folding achieves comparable performance to data-driven compression techniques and outperforms recently proposed data-free methods, especially at high sparsity levels. This approach is particularly effective for compressing large-scale models, making it suitable for deployment in resource-constrained environments. |
| title | Forget the Data and Fine-Tuning! Just Fold the Network to Compress |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2502.10216 |