Updating CLIP to Prefer Descriptions Over Captions
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913529892700160 |
|---|---|
| author | Zur, Amir Kreiss, Elisa D'Oosterlinck, Karel Potts, Christopher Geiger, Atticus |
| author_facet | Zur, Amir Kreiss, Elisa D'Oosterlinck, Karel Potts, Christopher Geiger, Atticus |
| contents | Although CLIPScore is a powerful generic metric that captures the similarity between a text and an image, it fails to distinguish between a caption that is meant to complement the information in an image and a description that is meant to replace an image entirely, e.g., for accessibility. We address this shortcoming by updating the CLIP model with the Concadia dataset to assign higher scores to descriptions than captions using parameter efficient fine-tuning and a loss objective derived from work on causal interpretability. This model correlates with the judgements of blind and low-vision people while preserving transfer capabilities and has interpretable structure that sheds light on the caption--description distinction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_09458 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Updating CLIP to Prefer Descriptions Over Captions Zur, Amir Kreiss, Elisa D'Oosterlinck, Karel Potts, Christopher Geiger, Atticus Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Although CLIPScore is a powerful generic metric that captures the similarity between a text and an image, it fails to distinguish between a caption that is meant to complement the information in an image and a description that is meant to replace an image entirely, e.g., for accessibility. We address this shortcoming by updating the CLIP model with the Concadia dataset to assign higher scores to descriptions than captions using parameter efficient fine-tuning and a loss objective derived from work on causal interpretability. This model correlates with the judgements of blind and low-vision people while preserving transfer capabilities and has interpretable structure that sheds light on the caption--description distinction. |
| title | Updating CLIP to Prefer Descriptions Over Captions |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2406.09458 |