Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.19196 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918126342373376 |
|---|---|
| author | Janssens, Ruben Belpaeme, Tony |
| author_facet | Janssens, Ruben Belpaeme, Tony |
| contents | Large language models have given social robots the ability to autonomously engage in open-domain conversations. However, they are still missing a fundamental social skill: making use of the multiple modalities that carry social interactions. While previous work has focused on task-oriented interactions that require referencing the environment or specific phenomena in social interactions such as dialogue breakdowns, we outline the overall needs of a multimodal system for social conversations with robots. We then argue that vision-language models are able to process this wide range of visual information in a sufficiently general manner for autonomous social robots. We describe how to adapt them to this setting, which technical challenges remain, and briefly discuss evaluation practices. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_19196 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards Multimodal Social Conversations with Robots: Using Vision-Language Models Janssens, Ruben Belpaeme, Tony Robotics Computation and Language Human-Computer Interaction Large language models have given social robots the ability to autonomously engage in open-domain conversations. However, they are still missing a fundamental social skill: making use of the multiple modalities that carry social interactions. While previous work has focused on task-oriented interactions that require referencing the environment or specific phenomena in social interactions such as dialogue breakdowns, we outline the overall needs of a multimodal system for social conversations with robots. We then argue that vision-language models are able to process this wide range of visual information in a sufficiently general manner for autonomous social robots. We describe how to adapt them to this setting, which technical challenges remain, and briefly discuss evaluation practices. |
| title | Towards Multimodal Social Conversations with Robots: Using Vision-Language Models |
| topic | Robotics Computation and Language Human-Computer Interaction |
| url | https://arxiv.org/abs/2507.19196 |