Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfer
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915303487700992 |
|---|---|
| author | Du, Guodong Fang, Zitao Li, Jing Li, Junlin Jiang, Runhua Yu, Shuyang Guo, Yifei Chen, Yangneng Goh, Sim Kuan Tang, Ho-Kin He, Daojing Liu, Honghai Zhang, Min |
| author_facet | Du, Guodong Fang, Zitao Li, Jing Li, Junlin Jiang, Runhua Yu, Shuyang Guo, Yifei Chen, Yangneng Goh, Sim Kuan Tang, Ho-Kin He, Daojing Liu, Honghai Zhang, Min |
| contents | Foundation models and their checkpoints have significantly advanced deep learning, boosting performance across various applications. However, fine-tuned models often struggle outside their specific domains and exhibit considerable redundancy. Recent studies suggest that combining a pruned fine-tuned model with the original pre-trained model can mitigate forgetting, reduce interference when merging model parameters across tasks, and improve compression efficiency. In this context, developing an effective pruning strategy for fine-tuned models is crucial. Leveraging the advantages of the task vector mechanism, we preprocess fine-tuned models by calculating the differences between them and the original model. Recognizing that different task vector subspaces contribute variably to model performance, we introduce a novel method called Neural Parameter Search (NPS-Pruning) for slimming down fine-tuned models. This method enhances pruning efficiency by searching through neural parameters of task vectors within low-rank subspaces. Our method has three key applications: enhancing knowledge transfer through pairwise model interpolation, facilitating effective knowledge fusion via model merging, and enabling the deployment of compressed models that retain near-original performance while significantly reducing storage costs. Extensive experiments across vision, NLP, and multi-modal benchmarks demonstrate the effectiveness and robustness of our approach, resulting in substantial performance gains. The code is publicly available at: https://github.com/duguodong7/NPS-Pruning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_18713 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfer Du, Guodong Fang, Zitao Li, Jing Li, Junlin Jiang, Runhua Yu, Shuyang Guo, Yifei Chen, Yangneng Goh, Sim Kuan Tang, Ho-Kin He, Daojing Liu, Honghai Zhang, Min Machine Learning Artificial Intelligence Computation and Language Foundation models and their checkpoints have significantly advanced deep learning, boosting performance across various applications. However, fine-tuned models often struggle outside their specific domains and exhibit considerable redundancy. Recent studies suggest that combining a pruned fine-tuned model with the original pre-trained model can mitigate forgetting, reduce interference when merging model parameters across tasks, and improve compression efficiency. In this context, developing an effective pruning strategy for fine-tuned models is crucial. Leveraging the advantages of the task vector mechanism, we preprocess fine-tuned models by calculating the differences between them and the original model. Recognizing that different task vector subspaces contribute variably to model performance, we introduce a novel method called Neural Parameter Search (NPS-Pruning) for slimming down fine-tuned models. This method enhances pruning efficiency by searching through neural parameters of task vectors within low-rank subspaces. Our method has three key applications: enhancing knowledge transfer through pairwise model interpolation, facilitating effective knowledge fusion via model merging, and enabling the deployment of compressed models that retain near-original performance while significantly reducing storage costs. Extensive experiments across vision, NLP, and multi-modal benchmarks demonstrate the effectiveness and robustness of our approach, resulting in substantial performance gains. The code is publicly available at: https://github.com/duguodong7/NPS-Pruning. |
| title | Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfer |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2505.18713 |