Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.06990 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912576678395904 |
|---|---|
| author | Rodas, Bryan Montesino, Natalie Ambsdorf, Jakob Klindt, David Balestriero, Randall |
| author_facet | Rodas, Bryan Montesino, Natalie Ambsdorf, Jakob Klindt, David Balestriero, Randall |
| contents | Continued pretraining offers a promising solution for adapting foundation models to a new target domain. However, in specialized domains, available datasets are often very small, limiting the applicability of SSL methods developed for large-scale pretraining and making hyperparameter search infeasible. In addition, pretrained models are usually released as backbone-weights only, lacking important information to continue pretraining. We propose to bridge this gap with DIET-CP, a simple continued pretraining strategy, where any strong foundation model can be steered towards the new data distribution of interest. DIET-CP relies on a very simple objective, requires no labels, and introduces no more hyperparameters than supervised finetuning. It is stable across data modalities and backbone choices, while providing a significant performance boost for state-of-the-art models such as DINOv3 using only 1000 images. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_06990 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DIET-CP: Lightweight and Data Efficient Self Supervised Continued Pretraining Rodas, Bryan Montesino, Natalie Ambsdorf, Jakob Klindt, David Balestriero, Randall Computer Vision and Pattern Recognition Machine Learning I.2; I.4 Continued pretraining offers a promising solution for adapting foundation models to a new target domain. However, in specialized domains, available datasets are often very small, limiting the applicability of SSL methods developed for large-scale pretraining and making hyperparameter search infeasible. In addition, pretrained models are usually released as backbone-weights only, lacking important information to continue pretraining. We propose to bridge this gap with DIET-CP, a simple continued pretraining strategy, where any strong foundation model can be steered towards the new data distribution of interest. DIET-CP relies on a very simple objective, requires no labels, and introduces no more hyperparameters than supervised finetuning. It is stable across data modalities and backbone choices, while providing a significant performance boost for state-of-the-art models such as DINOv3 using only 1000 images. |
| title | DIET-CP: Lightweight and Data Efficient Self Supervised Continued Pretraining |
| topic | Computer Vision and Pattern Recognition Machine Learning I.2; I.4 |
| url | https://arxiv.org/abs/2509.06990 |