Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2505.21835 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912399384117248 |
|---|---|
| author | Chen, Xiangyu Liu, Jing Wang, Ye Brand, Matthew Pu Wang Koike-Akino, Toshiaki |
| author_facet | Chen, Xiangyu Liu, Jing Wang, Ye Brand, Matthew Pu Wang Koike-Akino, Toshiaki |
| contents | To reduce model size during post-training, compression methods, including knowledge distillation, low-rank approximation, and pruning, are often applied after fine-tuning the model. However, sequential fine-tuning and compression sacrifices performance, while creating a larger than necessary model as an intermediate step. In this work, we aim to reduce this gap, by directly constructing a smaller model while guided by the downstream task. We propose to jointly fine-tune and compress the model by gradually distilling it to a pruned low-rank structure. Experiments demonstrate that joint fine-tuning and compression significantly outperforms other sequential compression methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_21835 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TuneComp: Joint Fine-tuning and Compression for Large Foundation Models Chen, Xiangyu Liu, Jing Wang, Ye Brand, Matthew Pu Wang Koike-Akino, Toshiaki Machine Learning Artificial Intelligence To reduce model size during post-training, compression methods, including knowledge distillation, low-rank approximation, and pruning, are often applied after fine-tuning the model. However, sequential fine-tuning and compression sacrifices performance, while creating a larger than necessary model as an intermediate step. In this work, we aim to reduce this gap, by directly constructing a smaller model while guided by the downstream task. We propose to jointly fine-tune and compress the model by gradually distilling it to a pruned low-rank structure. Experiments demonstrate that joint fine-tuning and compression significantly outperforms other sequential compression methods. |
| title | TuneComp: Joint Fine-tuning and Compression for Large Foundation Models |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2505.21835 |