An Empirical Study of Pre-trained Model Selection for Out-of-Distribution Generalization and Calibration
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913808756244480 |
|---|---|
| author | Naganuma, Hiroki Hataya, Ryuichiro Yoshida, Kotaro Mitliagkas, Ioannis |
| author_facet | Naganuma, Hiroki Hataya, Ryuichiro Yoshida, Kotaro Mitliagkas, Ioannis |
| contents | In the field of computer vision, fine-tuning pre-trained models has become a prevalent strategy for out-of-distribution (OOD) generalization tasks. Different from most prior work that has focused on advancing learning algorithms, we systematically examined how pre-trained model size, pre-training dataset size, and training strategies impact generalization and confidence calibration on downstream tasks. We evaluated 100 models across diverse pre-trained model sizes, five pre-training datasets, and five data augmentations through extensive experiments on four distribution shift datasets totaling over 120,000 GPU hours. Our results demonstrate the significant impact of pre-trained model selection, with optimal choices substantially improving OOD accuracy over algorithm improvement alone. Additionally, we find that larger models and bigger pre-training datasets not only enhance OOD performance but also improve calibration, helping to mitigate overconfidence, contrary to some prior studies that found modern deep networks to calibrate worse than classical shallow models. Our work underscores the overlooked importance of pre-trained model selection for out-of-distribution generalization and calibration. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2307_08187 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | An Empirical Study of Pre-trained Model Selection for Out-of-Distribution Generalization and Calibration Naganuma, Hiroki Hataya, Ryuichiro Yoshida, Kotaro Mitliagkas, Ioannis Machine Learning Artificial Intelligence In the field of computer vision, fine-tuning pre-trained models has become a prevalent strategy for out-of-distribution (OOD) generalization tasks. Different from most prior work that has focused on advancing learning algorithms, we systematically examined how pre-trained model size, pre-training dataset size, and training strategies impact generalization and confidence calibration on downstream tasks. We evaluated 100 models across diverse pre-trained model sizes, five pre-training datasets, and five data augmentations through extensive experiments on four distribution shift datasets totaling over 120,000 GPU hours. Our results demonstrate the significant impact of pre-trained model selection, with optimal choices substantially improving OOD accuracy over algorithm improvement alone. Additionally, we find that larger models and bigger pre-training datasets not only enhance OOD performance but also improve calibration, helping to mitigate overconfidence, contrary to some prior studies that found modern deep networks to calibrate worse than classical shallow models. Our work underscores the overlooked importance of pre-trained model selection for out-of-distribution generalization and calibration. |
| title | An Empirical Study of Pre-trained Model Selection for Out-of-Distribution Generalization and Calibration |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2307.08187 |