WATT: Weight Average Test-Time Adaptation of CLIP
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910499973627904 |
|---|---|
| author | Osowiechi, David Noori, Mehrdad Hakim, Gustavo Adolfo Vargas Yazdanpanah, Moslem Bahri, Ali Cheraghalikhani, Milad Dastani, Sahar Beizaee, Farzad Ayed, Ismail Ben Desrosiers, Christian |
| author_facet | Osowiechi, David Noori, Mehrdad Hakim, Gustavo Adolfo Vargas Yazdanpanah, Moslem Bahri, Ali Cheraghalikhani, Milad Dastani, Sahar Beizaee, Farzad Ayed, Ismail Ben Desrosiers, Christian |
| contents | Vision-Language Models (VLMs) such as CLIP have yielded unprecedented performance for zero-shot image classification, yet their generalization capability may still be seriously challenged when confronted to domain shifts. In response, we present Weight Average Test-Time Adaptation (WATT) of CLIP, a pioneering approach facilitating full test-time adaptation (TTA) of this VLM. Our method employs a diverse set of templates for text prompts, augmenting the existing framework of CLIP. Predictions are utilized as pseudo labels for model updates, followed by weight averaging to consolidate the learned information globally. Furthermore, we introduce a text ensemble strategy, enhancing overall test performance by aggregating diverse textual cues. Our findings underscore the efficacy of WATT in enhancing performance across diverse datasets, including CIFAR-10-C, CIFAR-10.1, CIFAR-100-C, VisDA-C, and several other challenging datasets, effectively covering a wide range of domain shifts. Notably, these enhancements are achieved without necessitating additional model transformations or trainable modules. Moreover, compared to other Test-Time Adaptation methods, our approach can operate effectively with just a single image. Highlighting the potential of innovative test-time strategies, this research emphasizes their role in fortifying the adaptability of VLMs. The implementation is available at: \url{https://github.com/Mehrdad-Noori/WATT.git}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_13875 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | WATT: Weight Average Test-Time Adaptation of CLIP Osowiechi, David Noori, Mehrdad Hakim, Gustavo Adolfo Vargas Yazdanpanah, Moslem Bahri, Ali Cheraghalikhani, Milad Dastani, Sahar Beizaee, Farzad Ayed, Ismail Ben Desrosiers, Christian Computer Vision and Pattern Recognition Vision-Language Models (VLMs) such as CLIP have yielded unprecedented performance for zero-shot image classification, yet their generalization capability may still be seriously challenged when confronted to domain shifts. In response, we present Weight Average Test-Time Adaptation (WATT) of CLIP, a pioneering approach facilitating full test-time adaptation (TTA) of this VLM. Our method employs a diverse set of templates for text prompts, augmenting the existing framework of CLIP. Predictions are utilized as pseudo labels for model updates, followed by weight averaging to consolidate the learned information globally. Furthermore, we introduce a text ensemble strategy, enhancing overall test performance by aggregating diverse textual cues. Our findings underscore the efficacy of WATT in enhancing performance across diverse datasets, including CIFAR-10-C, CIFAR-10.1, CIFAR-100-C, VisDA-C, and several other challenging datasets, effectively covering a wide range of domain shifts. Notably, these enhancements are achieved without necessitating additional model transformations or trainable modules. Moreover, compared to other Test-Time Adaptation methods, our approach can operate effectively with just a single image. Highlighting the potential of innovative test-time strategies, this research emphasizes their role in fortifying the adaptability of VLMs. The implementation is available at: \url{https://github.com/Mehrdad-Noori/WATT.git}. |
| title | WATT: Weight Average Test-Time Adaptation of CLIP |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.13875 |