Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2406.14035 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929623404642304 |
|---|---|
| author | Hakimov, Sherzod Abdullayeva, Yerkezhan Koshti, Kushal Schmidt, Antonia Weiser, Yan Beyer, Anne Schlangen, David |
| author_facet | Hakimov, Sherzod Abdullayeva, Yerkezhan Koshti, Kushal Schmidt, Antonia Weiser, Yan Beyer, Anne Schlangen, David |
| contents | While the situation has improved for text-only models, it again seems to be the case currently that multimodal (text and image) models develop faster than ways to evaluate them. In this paper, we bring a recently developed evaluation paradigm from text models to multimodal models, namely evaluation through the goal-oriented game (self) play, complementing reference-based and preference-based evaluation. Specifically, we define games that challenge a model's capability to represent a situation from visual information and align such representations through dialogue. We find that the largest closed models perform rather well on the games that we define, while even the best open-weight models struggle with them. On further analysis, we find that the exceptional deep captioning capabilities of the largest models drive some of the performance. There is still room to grow for both kinds of models, ensuring the continued relevance of the benchmark. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_14035 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Using Game Play to Investigate Multimodal and Conversational Grounding in Large Multimodal Models Hakimov, Sherzod Abdullayeva, Yerkezhan Koshti, Kushal Schmidt, Antonia Weiser, Yan Beyer, Anne Schlangen, David Computation and Language Artificial Intelligence While the situation has improved for text-only models, it again seems to be the case currently that multimodal (text and image) models develop faster than ways to evaluate them. In this paper, we bring a recently developed evaluation paradigm from text models to multimodal models, namely evaluation through the goal-oriented game (self) play, complementing reference-based and preference-based evaluation. Specifically, we define games that challenge a model's capability to represent a situation from visual information and align such representations through dialogue. We find that the largest closed models perform rather well on the games that we define, while even the best open-weight models struggle with them. On further analysis, we find that the exceptional deep captioning capabilities of the largest models drive some of the performance. There is still room to grow for both kinds of models, ensuring the continued relevance of the benchmark. |
| title | Using Game Play to Investigate Multimodal and Conversational Grounding in Large Multimodal Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2406.14035 |