To Stay or Not to Stay in the Pre-train Basin: Insights on Ensembling in Transfer Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910296981897216 |
|---|---|
| author | Sadrtdinov, Ildus Pozdeev, Dmitrii Vetrov, Dmitry Lobacheva, Ekaterina |
| author_facet | Sadrtdinov, Ildus Pozdeev, Dmitrii Vetrov, Dmitry Lobacheva, Ekaterina |
| contents | Transfer learning and ensembling are two popular techniques for improving the performance and robustness of neural networks. Due to the high cost of pre-training, ensembles of models fine-tuned from a single pre-trained checkpoint are often used in practice. Such models end up in the same basin of the loss landscape, which we call the pre-train basin, and thus have limited diversity. In this work, we show that ensembles trained from a single pre-trained checkpoint may be improved by better exploring the pre-train basin, however, leaving the basin results in losing the benefits of transfer learning and in degradation of the ensemble quality. Based on the analysis of existing exploration methods, we propose a more effective modification of the Snapshot Ensembles (SSE) for transfer learning setup, StarSSE, which results in stronger ensembles and uniform model soups. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2303_03374 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | To Stay or Not to Stay in the Pre-train Basin: Insights on Ensembling in Transfer Learning Sadrtdinov, Ildus Pozdeev, Dmitrii Vetrov, Dmitry Lobacheva, Ekaterina Machine Learning Transfer learning and ensembling are two popular techniques for improving the performance and robustness of neural networks. Due to the high cost of pre-training, ensembles of models fine-tuned from a single pre-trained checkpoint are often used in practice. Such models end up in the same basin of the loss landscape, which we call the pre-train basin, and thus have limited diversity. In this work, we show that ensembles trained from a single pre-trained checkpoint may be improved by better exploring the pre-train basin, however, leaving the basin results in losing the benefits of transfer learning and in degradation of the ensemble quality. Based on the analysis of existing exploration methods, we propose a more effective modification of the Snapshot Ensembles (SSE) for transfer learning setup, StarSSE, which results in stronger ensembles and uniform model soups. |
| title | To Stay or Not to Stay in the Pre-train Basin: Insights on Ensembling in Transfer Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2303.03374 |