Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911897654132736 |
|---|---|
| author | Shi, Jiang-Xin Wei, Tong Zhou, Zhi Shao, Jie-Jing Han, Xin-Yan Li, Yu-Feng |
| author_facet | Shi, Jiang-Xin Wei, Tong Zhou, Zhi Shao, Jie-Jing Han, Xin-Yan Li, Yu-Feng |
| contents | The fine-tuning paradigm in addressing long-tail learning tasks has sparked significant interest since the emergence of foundation models. Nonetheless, how fine-tuning impacts performance in long-tail learning was not explicitly quantified. In this paper, we disclose that heavy fine-tuning may even lead to non-negligible performance deterioration on tail classes, and lightweight fine-tuning is more effective. The reason is attributed to inconsistent class conditions caused by heavy fine-tuning. With the observation above, we develop a low-complexity and accurate long-tail learning algorithms LIFT with the goal of facilitating fast prediction and compact models by adaptive lightweight fine-tuning. Experiments clearly verify that both the training time and the learned parameters are significantly reduced with more accurate predictive performance compared with state-of-the-art approaches. The implementation code is available at https://github.com/shijxcs/LIFT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_10019 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts Shi, Jiang-Xin Wei, Tong Zhou, Zhi Shao, Jie-Jing Han, Xin-Yan Li, Yu-Feng Computer Vision and Pattern Recognition Machine Learning The fine-tuning paradigm in addressing long-tail learning tasks has sparked significant interest since the emergence of foundation models. Nonetheless, how fine-tuning impacts performance in long-tail learning was not explicitly quantified. In this paper, we disclose that heavy fine-tuning may even lead to non-negligible performance deterioration on tail classes, and lightweight fine-tuning is more effective. The reason is attributed to inconsistent class conditions caused by heavy fine-tuning. With the observation above, we develop a low-complexity and accurate long-tail learning algorithms LIFT with the goal of facilitating fast prediction and compact models by adaptive lightweight fine-tuning. Experiments clearly verify that both the training time and the learned parameters are significantly reduced with more accurate predictive performance compared with state-of-the-art approaches. The implementation code is available at https://github.com/shijxcs/LIFT. |
| title | Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2309.10019 |