Learning from Offline Foundation Features with Tensor Augmentations
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914963563479040 |
|---|---|
| author | Konuk, Emir Matsoukas, Christos Sorkhei, Moein Lertsiravaramet, Phitchapha Smith, Kevin |
| author_facet | Konuk, Emir Matsoukas, Christos Sorkhei, Moein Lertsiravaramet, Phitchapha Smith, Kevin |
| contents | We introduce Learning from Offline Foundation Features with Tensor Augmentations (LOFF-TA), an efficient training scheme designed to harness the capabilities of foundation models in limited resource settings where their direct development is not feasible. LOFF-TA involves training a compact classifier on cached feature embeddings from a frozen foundation model, resulting in up to $37\times$ faster training and up to $26\times$ reduced GPU memory usage. Because the embeddings of augmented images would be too numerous to store, yet the augmentation process is essential for training, we propose to apply tensor augmentations to the cached embeddings of the original non-augmented images. LOFF-TA makes it possible to leverage the power of foundation models, regardless of their size, in settings with limited computational capacity. Moreover, LOFF-TA can be used to apply foundation models to high-resolution images without increasing compute. In certain scenarios, we find that training with LOFF-TA yields better results than directly fine-tuning the foundation model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_02527 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Learning from Offline Foundation Features with Tensor Augmentations Konuk, Emir Matsoukas, Christos Sorkhei, Moein Lertsiravaramet, Phitchapha Smith, Kevin Computer Vision and Pattern Recognition We introduce Learning from Offline Foundation Features with Tensor Augmentations (LOFF-TA), an efficient training scheme designed to harness the capabilities of foundation models in limited resource settings where their direct development is not feasible. LOFF-TA involves training a compact classifier on cached feature embeddings from a frozen foundation model, resulting in up to $37\times$ faster training and up to $26\times$ reduced GPU memory usage. Because the embeddings of augmented images would be too numerous to store, yet the augmentation process is essential for training, we propose to apply tensor augmentations to the cached embeddings of the original non-augmented images. LOFF-TA makes it possible to leverage the power of foundation models, regardless of their size, in settings with limited computational capacity. Moreover, LOFF-TA can be used to apply foundation models to high-resolution images without increasing compute. In certain scenarios, we find that training with LOFF-TA yields better results than directly fine-tuning the foundation model. |
| title | Learning from Offline Foundation Features with Tensor Augmentations |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.02527 |