CARMA: Context-Aware Situational Grounding of Human-Robot Group Interactions by Combining Vision-Language Models with Object and Action Recognition
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913912198266880 |
|---|---|
| author | Deigmoeller, Joerg Hasler, Stephan Agarwal, Nakul Tanneberg, Daniel Belardinelli, Anna Ghoddoosian, Reza Wang, Chao Ocker, Felix Zhang, Fan Dariush, Behzad Gienger, Michael |
| author_facet | Deigmoeller, Joerg Hasler, Stephan Agarwal, Nakul Tanneberg, Daniel Belardinelli, Anna Ghoddoosian, Reza Wang, Chao Ocker, Felix Zhang, Fan Dariush, Behzad Gienger, Michael |
| contents | We introduce CARMA, a system for situational grounding in human-robot group interactions. Effective collaboration in such group settings requires situational awareness based on a consistent representation of present persons and objects coupled with an episodic abstraction of events regarding actors and manipulated objects. This calls for a clear and consistent assignment of instances, ensuring that robots correctly recognize and track actors, objects, and their interactions over time. To achieve this, CARMA uniquely identifies physical instances of such entities in the real world and organizes them into grounded triplets of actors, objects, and actions.
To validate our approach, we conducted three experiments, where multiple humans and a robot interact: collaborative pouring, handovers, and sorting. These scenarios allow the assessment of the system's capabilities as to role distinction, multi-actor awareness, and consistent instance identification. Our experiments demonstrate that the system can reliably generate accurate actor-action-object triplets, providing a structured and robust foundation for applications requiring spatiotemporal reasoning and situated decision-making in collaborative settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_20373 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CARMA: Context-Aware Situational Grounding of Human-Robot Group Interactions by Combining Vision-Language Models with Object and Action Recognition Deigmoeller, Joerg Hasler, Stephan Agarwal, Nakul Tanneberg, Daniel Belardinelli, Anna Ghoddoosian, Reza Wang, Chao Ocker, Felix Zhang, Fan Dariush, Behzad Gienger, Michael Robotics Artificial Intelligence Human-Computer Interaction We introduce CARMA, a system for situational grounding in human-robot group interactions. Effective collaboration in such group settings requires situational awareness based on a consistent representation of present persons and objects coupled with an episodic abstraction of events regarding actors and manipulated objects. This calls for a clear and consistent assignment of instances, ensuring that robots correctly recognize and track actors, objects, and their interactions over time. To achieve this, CARMA uniquely identifies physical instances of such entities in the real world and organizes them into grounded triplets of actors, objects, and actions. To validate our approach, we conducted three experiments, where multiple humans and a robot interact: collaborative pouring, handovers, and sorting. These scenarios allow the assessment of the system's capabilities as to role distinction, multi-actor awareness, and consistent instance identification. Our experiments demonstrate that the system can reliably generate accurate actor-action-object triplets, providing a structured and robust foundation for applications requiring spatiotemporal reasoning and situated decision-making in collaborative settings. |
| title | CARMA: Context-Aware Situational Grounding of Human-Robot Group Interactions by Combining Vision-Language Models with Object and Action Recognition |
| topic | Robotics Artificial Intelligence Human-Computer Interaction |
| url | https://arxiv.org/abs/2506.20373 |