FALIP: Visual Prompt as Foveal Attention Boosts CLIP Zero-Shot Performance
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911997262561280 |
|---|---|
| author | Zhuang, Jiedong Hu, Jiaqi Mu, Lianrui Hu, Rui Liang, Xiaoyu Ye, Jiangnan Hu, Haoji |
| author_facet | Zhuang, Jiedong Hu, Jiaqi Mu, Lianrui Hu, Rui Liang, Xiaoyu Ye, Jiangnan Hu, Haoji |
| contents | CLIP has achieved impressive zero-shot performance after pre-training on a large-scale dataset consisting of paired image-text data. Previous works have utilized CLIP by incorporating manually designed visual prompts like colored circles and blur masks into the images to guide the model's attention, showing enhanced zero-shot performance in downstream tasks. Although these methods have achieved promising results, they inevitably alter the original information of the images, which can lead to failure in specific tasks. We propose a train-free method Foveal-Attention CLIP (FALIP), which adjusts the CLIP's attention by inserting foveal attention masks into the multi-head self-attention module. We demonstrate FALIP effectively boosts CLIP zero-shot performance in tasks such as referring expressions comprehension, image classification, and 3D point cloud recognition. Experimental results further show that FALIP outperforms existing methods on most metrics and can augment current methods to enhance their performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_05578 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FALIP: Visual Prompt as Foveal Attention Boosts CLIP Zero-Shot Performance Zhuang, Jiedong Hu, Jiaqi Mu, Lianrui Hu, Rui Liang, Xiaoyu Ye, Jiangnan Hu, Haoji Computer Vision and Pattern Recognition CLIP has achieved impressive zero-shot performance after pre-training on a large-scale dataset consisting of paired image-text data. Previous works have utilized CLIP by incorporating manually designed visual prompts like colored circles and blur masks into the images to guide the model's attention, showing enhanced zero-shot performance in downstream tasks. Although these methods have achieved promising results, they inevitably alter the original information of the images, which can lead to failure in specific tasks. We propose a train-free method Foveal-Attention CLIP (FALIP), which adjusts the CLIP's attention by inserting foveal attention masks into the multi-head self-attention module. We demonstrate FALIP effectively boosts CLIP zero-shot performance in tasks such as referring expressions comprehension, image classification, and 3D point cloud recognition. Experimental results further show that FALIP outperforms existing methods on most metrics and can augment current methods to enhance their performance. |
| title | FALIP: Visual Prompt as Foveal Attention Boosts CLIP Zero-Shot Performance |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.05578 |