SemanticScanpath: Combining Gaze and Speech for Situated Human-Robot Interaction Using LLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910110784159744 |
|---|---|
| author | Menendez, Elisabeth Gienger, Michael Martínez, Santiago Balaguer, Carlos Belardinelli, Anna |
| author_facet | Menendez, Elisabeth Gienger, Michael Martínez, Santiago Balaguer, Carlos Belardinelli, Anna |
| contents | Large Language Models (LLMs) have substantially improved the conversational capabilities of social robots. Nevertheless, for an intuitive and fluent human-robot interaction, robots should be able to ground the conversation by relating ambiguous or underspecified spoken utterances to the current physical situation and to the intents expressed nonverbally by the user, such as through referential gaze. Here, we propose a representation that integrates speech and gaze to enable LLMs to achieve higher situated awareness and correctly resolve ambiguous requests. Our approach relies on a text-based semantic translation of the scanpath produced by the user, along with the verbal requests. It demonstrates LLMs' capabilities to reason about gaze behavior, robustly ignoring spurious glances or irrelevant objects. We validate the system across multiple tasks and two scenarios, showing its superior generality and accuracy compared to control conditions. We demonstrate an implementation on a robotic platform, closing the loop from request interpretation to execution. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_16548 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SemanticScanpath: Combining Gaze and Speech for Situated Human-Robot Interaction Using LLMs Menendez, Elisabeth Gienger, Michael Martínez, Santiago Balaguer, Carlos Belardinelli, Anna Human-Computer Interaction Robotics Large Language Models (LLMs) have substantially improved the conversational capabilities of social robots. Nevertheless, for an intuitive and fluent human-robot interaction, robots should be able to ground the conversation by relating ambiguous or underspecified spoken utterances to the current physical situation and to the intents expressed nonverbally by the user, such as through referential gaze. Here, we propose a representation that integrates speech and gaze to enable LLMs to achieve higher situated awareness and correctly resolve ambiguous requests. Our approach relies on a text-based semantic translation of the scanpath produced by the user, along with the verbal requests. It demonstrates LLMs' capabilities to reason about gaze behavior, robustly ignoring spurious glances or irrelevant objects. We validate the system across multiple tasks and two scenarios, showing its superior generality and accuracy compared to control conditions. We demonstrate an implementation on a robotic platform, closing the loop from request interpretation to execution. |
| title | SemanticScanpath: Combining Gaze and Speech for Situated Human-Robot Interaction Using LLMs |
| topic | Human-Computer Interaction Robotics |
| url | https://arxiv.org/abs/2503.16548 |