Caption-Driven Explorations: Aligning Image and Text Embeddings through Human-Inspired Foveated Vision
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911994364297216 |
|---|---|
| author | Zanca, Dario Zugarini, Andrea Dietz, Simon Altstidl, Thomas R. Ndjeuha, Mark A. Turban Schwinn, Leo Eskofier, Bjoern |
| author_facet | Zanca, Dario Zugarini, Andrea Dietz, Simon Altstidl, Thomas R. Ndjeuha, Mark A. Turban Schwinn, Leo Eskofier, Bjoern |
| contents | Understanding human attention is crucial for vision science and AI. While many models exist for free-viewing, less is known about task-driven image exploration. To address this, we introduce CapMIT1003, a dataset with captions and click-contingent image explorations, to study human attention during the captioning task. We also present NevaClip, a zero-shot method for predicting visual scanpaths by combining CLIP models with NeVA algorithms. NevaClip generates fixations to align the representations of foveated visual stimuli and captions. The simulated scanpaths outperform existing human attention models in plausibility for captioning and free-viewing tasks. This research enhances the understanding of human attention and advances scanpath prediction models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_09948 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Caption-Driven Explorations: Aligning Image and Text Embeddings through Human-Inspired Foveated Vision Zanca, Dario Zugarini, Andrea Dietz, Simon Altstidl, Thomas R. Ndjeuha, Mark A. Turban Schwinn, Leo Eskofier, Bjoern Computer Vision and Pattern Recognition Artificial Intelligence Understanding human attention is crucial for vision science and AI. While many models exist for free-viewing, less is known about task-driven image exploration. To address this, we introduce CapMIT1003, a dataset with captions and click-contingent image explorations, to study human attention during the captioning task. We also present NevaClip, a zero-shot method for predicting visual scanpaths by combining CLIP models with NeVA algorithms. NevaClip generates fixations to align the representations of foveated visual stimuli and captions. The simulated scanpaths outperform existing human attention models in plausibility for captioning and free-viewing tasks. This research enhances the understanding of human attention and advances scanpath prediction models. |
| title | Caption-Driven Explorations: Aligning Image and Text Embeddings through Human-Inspired Foveated Vision |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2408.09948 |