Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929603263594496 |
|---|---|
| author | Bitton-Guetta, Nitzan Slobodkin, Aviv Maimon, Aviya Habba, Eliya Rassin, Royi Bitton, Yonatan Szpektor, Idan Globerson, Amir Elovici, Yuval |
| author_facet | Bitton-Guetta, Nitzan Slobodkin, Aviv Maimon, Aviya Habba, Eliya Rassin, Royi Bitton, Yonatan Szpektor, Idan Globerson, Amir Elovici, Yuval |
| contents | Imagine observing someone scratching their arm; to understand why, additional context would be necessary. However, spotting a mosquito nearby would immediately offer a likely explanation for the person's discomfort, thereby alleviating the need for further information. This example illustrates how subtle visual cues can challenge our cognitive skills and demonstrates the complexity of interpreting visual scenarios. To study these skills, we present Visual Riddles, a benchmark aimed to test vision and language models on visual riddles requiring commonsense and world knowledge. The benchmark comprises 400 visual riddles, each featuring a unique image created by a variety of text-to-image models, question, ground-truth answer, textual hint, and attribution. Human evaluation reveals that existing models lag significantly behind human performance, which is at 82% accuracy, with Gemini-Pro-1.5 leading with 40% accuracy. Our benchmark comes with automatic evaluation tasks to make assessment scalable. These findings underscore the potential of Visual Riddles as a valuable resource for enhancing vision and language models' capabilities in interpreting complex visual scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_19474 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Models Bitton-Guetta, Nitzan Slobodkin, Aviv Maimon, Aviya Habba, Eliya Rassin, Royi Bitton, Yonatan Szpektor, Idan Globerson, Amir Elovici, Yuval Computer Vision and Pattern Recognition Computation and Language Imagine observing someone scratching their arm; to understand why, additional context would be necessary. However, spotting a mosquito nearby would immediately offer a likely explanation for the person's discomfort, thereby alleviating the need for further information. This example illustrates how subtle visual cues can challenge our cognitive skills and demonstrates the complexity of interpreting visual scenarios. To study these skills, we present Visual Riddles, a benchmark aimed to test vision and language models on visual riddles requiring commonsense and world knowledge. The benchmark comprises 400 visual riddles, each featuring a unique image created by a variety of text-to-image models, question, ground-truth answer, textual hint, and attribution. Human evaluation reveals that existing models lag significantly behind human performance, which is at 82% accuracy, with Gemini-Pro-1.5 leading with 40% accuracy. Our benchmark comes with automatic evaluation tasks to make assessment scalable. These findings underscore the potential of Visual Riddles as a valuable resource for enhancing vision and language models' capabilities in interpreting complex visual scenarios. |
| title | Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Models |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2407.19474 |