Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929206311518208 |
|---|---|
| author | Hassan, Jameel Gani, Hanan Hussein, Noor Khattak, Muhammad Uzair Naseer, Muzammal Khan, Fahad Shahbaz Khan, Salman |
| author_facet | Hassan, Jameel Gani, Hanan Hussein, Noor Khattak, Muhammad Uzair Naseer, Muzammal Khan, Fahad Shahbaz Khan, Salman |
| contents | The promising zero-shot generalization of vision-language models such as CLIP has led to their adoption using prompt learning for numerous downstream tasks. Previous works have shown test-time prompt tuning using entropy minimization to adapt text prompts for unseen domains. While effective, this overlooks the key cause for performance degradation to unseen domains -- distribution shift. In this work, we explicitly handle this problem by aligning the out-of-distribution (OOD) test sample statistics to those of the source data using prompt tuning. We use a single test sample to adapt multi-modal prompts at test time by minimizing the feature distribution shift to bridge the gap in the test domain. Evaluating against the domain generalization benchmark, our method improves zero-shot top- 1 accuracy beyond existing prompt-learning techniques, with a 3.08% improvement over the baseline MaPLe. In cross-dataset generalization with unseen categories across 10 datasets, our method improves consistently across all datasets compared to the existing state-of-the-art. Our source code and models are available at https://jameelhassan.github.io/promptalign. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_01459 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization Hassan, Jameel Gani, Hanan Hussein, Noor Khattak, Muhammad Uzair Naseer, Muzammal Khan, Fahad Shahbaz Khan, Salman Computer Vision and Pattern Recognition The promising zero-shot generalization of vision-language models such as CLIP has led to their adoption using prompt learning for numerous downstream tasks. Previous works have shown test-time prompt tuning using entropy minimization to adapt text prompts for unseen domains. While effective, this overlooks the key cause for performance degradation to unseen domains -- distribution shift. In this work, we explicitly handle this problem by aligning the out-of-distribution (OOD) test sample statistics to those of the source data using prompt tuning. We use a single test sample to adapt multi-modal prompts at test time by minimizing the feature distribution shift to bridge the gap in the test domain. Evaluating against the domain generalization benchmark, our method improves zero-shot top- 1 accuracy beyond existing prompt-learning techniques, with a 3.08% improvement over the baseline MaPLe. In cross-dataset generalization with unseen categories across 10 datasets, our method improves consistently across all datasets compared to the existing state-of-the-art. Our source code and models are available at https://jameelhassan.github.io/promptalign. |
| title | Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2311.01459 |