Specialized Foundation Models Struggle to Beat Supervised Baselines
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908276412645376 |
|---|---|
| author | Xu, Zongzhe Gupta, Ritvik Cheng, Wenduo Shen, Alexander Shen, Junhong Talwalkar, Ameet Khodak, Mikhail |
| author_facet | Xu, Zongzhe Gupta, Ritvik Cheng, Wenduo Shen, Alexander Shen, Junhong Talwalkar, Ameet Khodak, Mikhail |
| contents | Following its success for vision and text, the "foundation model" (FM) paradigm -- pretraining large models on massive data, then fine-tuning on target tasks -- has rapidly expanded to domains in the sciences, engineering, healthcare, and beyond. Has this achieved what the original FMs accomplished, i.e. the supplanting of traditional supervised learning in their domains? To answer we look at three modalities -- genomics, satellite imaging, and time series -- with multiple recent FMs and compare them to a standard supervised learning workflow: model development, hyperparameter tuning, and training, all using only data from the target task. Across these three specialized domains, we find that it is consistently possible to train simple supervised models -- no more complicated than a lightly modified wide ResNet or UNet -- that match or even outperform the latest foundation models. Our work demonstrates that the benefits of large-scale pretraining have yet to be realized in many specialized areas, reinforces the need to compare new FMs to strong, well-tuned baselines, and introduces two new, easy-to-use, open-source, and automated workflows for doing so. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_02796 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Specialized Foundation Models Struggle to Beat Supervised Baselines Xu, Zongzhe Gupta, Ritvik Cheng, Wenduo Shen, Alexander Shen, Junhong Talwalkar, Ameet Khodak, Mikhail Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Genomics Following its success for vision and text, the "foundation model" (FM) paradigm -- pretraining large models on massive data, then fine-tuning on target tasks -- has rapidly expanded to domains in the sciences, engineering, healthcare, and beyond. Has this achieved what the original FMs accomplished, i.e. the supplanting of traditional supervised learning in their domains? To answer we look at three modalities -- genomics, satellite imaging, and time series -- with multiple recent FMs and compare them to a standard supervised learning workflow: model development, hyperparameter tuning, and training, all using only data from the target task. Across these three specialized domains, we find that it is consistently possible to train simple supervised models -- no more complicated than a lightly modified wide ResNet or UNet -- that match or even outperform the latest foundation models. Our work demonstrates that the benefits of large-scale pretraining have yet to be realized in many specialized areas, reinforces the need to compare new FMs to strong, well-tuned baselines, and introduces two new, easy-to-use, open-source, and automated workflows for doing so. |
| title | Specialized Foundation Models Struggle to Beat Supervised Baselines |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Genomics |
| url | https://arxiv.org/abs/2411.02796 |