SAFT: Towards Out-of-Distribution Generalization in Fine-Tuning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910511522643968 |
|---|---|
| author | Nguyen, Bac Uhlich, Stefan Cardinaux, Fabien Mauch, Lukas Edraki, Marzieh Courville, Aaron |
| author_facet | Nguyen, Bac Uhlich, Stefan Cardinaux, Fabien Mauch, Lukas Edraki, Marzieh Courville, Aaron |
| contents | Handling distribution shifts from training data, known as out-of-distribution (OOD) generalization, poses a significant challenge in the field of machine learning. While a pre-trained vision-language model like CLIP has demonstrated remarkable zero-shot performance, further adaptation of the model to downstream tasks leads to undesirable degradation for OOD data. In this work, we introduce Sparse Adaptation for Fine-Tuning (SAFT), a method that prevents fine-tuning from forgetting the general knowledge in the pre-trained model. SAFT only updates a small subset of important parameters whose gradient magnitude is large, while keeping the other parameters frozen. SAFT is straightforward to implement and conceptually simple. Extensive experiments show that with only 0.1% of the model parameters, SAFT can significantly improve the performance of CLIP. It consistently outperforms baseline methods across several benchmarks. On the few-shot learning benchmark of ImageNet and its variants, SAFT gives a gain of 5.15% on average over the conventional fine-tuning method in OOD settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_03036 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SAFT: Towards Out-of-Distribution Generalization in Fine-Tuning Nguyen, Bac Uhlich, Stefan Cardinaux, Fabien Mauch, Lukas Edraki, Marzieh Courville, Aaron Computer Vision and Pattern Recognition Handling distribution shifts from training data, known as out-of-distribution (OOD) generalization, poses a significant challenge in the field of machine learning. While a pre-trained vision-language model like CLIP has demonstrated remarkable zero-shot performance, further adaptation of the model to downstream tasks leads to undesirable degradation for OOD data. In this work, we introduce Sparse Adaptation for Fine-Tuning (SAFT), a method that prevents fine-tuning from forgetting the general knowledge in the pre-trained model. SAFT only updates a small subset of important parameters whose gradient magnitude is large, while keeping the other parameters frozen. SAFT is straightforward to implement and conceptually simple. Extensive experiments show that with only 0.1% of the model parameters, SAFT can significantly improve the performance of CLIP. It consistently outperforms baseline methods across several benchmarks. On the few-shot learning benchmark of ImageNet and its variants, SAFT gives a gain of 5.15% on average over the conventional fine-tuning method in OOD settings. |
| title | SAFT: Towards Out-of-Distribution Generalization in Fine-Tuning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.03036 |