LVLMs are Bad at Overhearing Human Referential Communication
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911228953100288 |
|---|---|
| author | Wang, Zhengxiang Li, Weiling Kaliosis, Panagiotis Rambow, Owen Brennan, Susan E. |
| author_facet | Wang, Zhengxiang Li, Weiling Kaliosis, Panagiotis Rambow, Owen Brennan, Susan E. |
| contents | During spontaneous conversations, speakers collaborate on novel referring expressions, which they can then re-use in subsequent conversations. Understanding such referring expressions is an important ability for an embodied agent, so that it can carry out tasks in the real world. This requires integrating and understanding language, vision, and conversational interaction. We study the capabilities of seven state-of-the-art Large Vision Language Models (LVLMs) as overhearers to a corpus of spontaneous conversations between pairs of human discourse participants engaged in a collaborative object-matching task. We find that such a task remains challenging for current LVLMs and they all fail to show a consistent performance improvement as they overhear more conversations from the same discourse participants repeating the same task for multiple rounds. We release our corpus and code for reproducibility and to facilitate future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_11514 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LVLMs are Bad at Overhearing Human Referential Communication Wang, Zhengxiang Li, Weiling Kaliosis, Panagiotis Rambow, Owen Brennan, Susan E. Computation and Language During spontaneous conversations, speakers collaborate on novel referring expressions, which they can then re-use in subsequent conversations. Understanding such referring expressions is an important ability for an embodied agent, so that it can carry out tasks in the real world. This requires integrating and understanding language, vision, and conversational interaction. We study the capabilities of seven state-of-the-art Large Vision Language Models (LVLMs) as overhearers to a corpus of spontaneous conversations between pairs of human discourse participants engaged in a collaborative object-matching task. We find that such a task remains challenging for current LVLMs and they all fail to show a consistent performance improvement as they overhear more conversations from the same discourse participants repeating the same task for multiple rounds. We release our corpus and code for reproducibility and to facilitate future research. |
| title | LVLMs are Bad at Overhearing Human Referential Communication |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2509.11514 |