LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Views
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913397231058944 |
|---|---|
| author | Roh, Yuji Liu, Qingyun Gui, Huan Yuan, Zhe Tang, Yujin Whang, Steven Euijong Liu, Liang Bi, Shuchao Hong, Lichan Chi, Ed H. Zhao, Zhe |
| author_facet | Roh, Yuji Liu, Qingyun Gui, Huan Yuan, Zhe Tang, Yujin Whang, Steven Euijong Liu, Liang Bi, Shuchao Hong, Lichan Chi, Ed H. Zhao, Zhe |
| contents | Fine-tuning is becoming widely used for leveraging the power of pre-trained foundation models in new downstream tasks. While there are many successes of fine-tuning on various tasks, recent studies have observed challenges in the generalization of fine-tuned models to unseen distributions (i.e., out-of-distribution; OOD). To improve OOD generalization, some previous studies identify the limitations of fine-tuning data and regulate fine-tuning to preserve the general representation learned from pre-training data. However, potential limitations in the pre-training data and models are often ignored. In this paper, we contend that overly relying on the pre-trained representation may hinder fine-tuning from learning essential representations for downstream tasks and thus hurt its OOD generalization. It can be especially catastrophic when new tasks are from different (sub)domains compared to pre-training data. To address the issues in both pre-training and fine-tuning data, we propose a novel generalizable fine-tuning method LEVI (Layer-wise Ensemble of different VIews), where the pre-trained model is adaptively ensembled layer-wise with a small task-specific model, while preserving its efficiencies. By combining two complementing models, LEVI effectively suppresses problematic features in both the fine-tuning data and pre-trained model and preserves useful features for new tasks. Broad experiments with large language and vision models show that LEVI greatly improves fine-tuning generalization via emphasizing different views from fine-tuning data and pre-trained features. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_04644 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Views Roh, Yuji Liu, Qingyun Gui, Huan Yuan, Zhe Tang, Yujin Whang, Steven Euijong Liu, Liang Bi, Shuchao Hong, Lichan Chi, Ed H. Zhao, Zhe Machine Learning Artificial Intelligence Fine-tuning is becoming widely used for leveraging the power of pre-trained foundation models in new downstream tasks. While there are many successes of fine-tuning on various tasks, recent studies have observed challenges in the generalization of fine-tuned models to unseen distributions (i.e., out-of-distribution; OOD). To improve OOD generalization, some previous studies identify the limitations of fine-tuning data and regulate fine-tuning to preserve the general representation learned from pre-training data. However, potential limitations in the pre-training data and models are often ignored. In this paper, we contend that overly relying on the pre-trained representation may hinder fine-tuning from learning essential representations for downstream tasks and thus hurt its OOD generalization. It can be especially catastrophic when new tasks are from different (sub)domains compared to pre-training data. To address the issues in both pre-training and fine-tuning data, we propose a novel generalizable fine-tuning method LEVI (Layer-wise Ensemble of different VIews), where the pre-trained model is adaptively ensembled layer-wise with a small task-specific model, while preserving its efficiencies. By combining two complementing models, LEVI effectively suppresses problematic features in both the fine-tuning data and pre-trained model and preserves useful features for new tasks. Broad experiments with large language and vision models show that LEVI greatly improves fine-tuning generalization via emphasizing different views from fine-tuning data and pre-trained features. |
| title | LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Views |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2402.04644 |