Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909457549623296 |
|---|---|
| author | Payandeh, Amirreza Song, Daeun Nazeri, Mohammad Liang, Jing Mukherjee, Praneel Raj, Amir Hossain Kong, Yangzhe Manocha, Dinesh Xiao, Xuesu |
| author_facet | Payandeh, Amirreza Song, Daeun Nazeri, Mohammad Liang, Jing Mukherjee, Praneel Raj, Amir Hossain Kong, Yangzhe Manocha, Dinesh Xiao, Xuesu |
| contents | Most existing social robot navigation techniques either leverage hand-crafted rules or human demonstrations to connect robot perception to socially compliant actions. However, there remains a significant gap in effectively translating perception into socially compliant actions, much like how human reasoning naturally occurs in dynamic environments. Considering the recent success of Vision-Language Models (VLMs), we propose using language to bridge the gap in human-like reasoning between perception and socially aware robot actions. We create a vision-language dataset, Social robot Navigation via Explainable Interactions (SNEI), featuring 40K human-annotated Visual Question Answers (VQAs) based on 2K human-robot social interactions in unstructured, crowded public spaces, spanning perception, prediction, chain-of-thought reasoning, action, and explanation. We fine-tune a VLM, Social-LLaVA, using SNEI to demonstrate the practical application of our dataset. Social-LLaVA outperforms state-of-the-art models like GPT-4V and Gemini, based on the average of fifteen different human-judge scores across 50 VQA. Deployed onboard a mobile robot, Social-LLaVA enables human-like reasoning, marking a promising step toward socially compliant robot navigation in dynamic public spaces through language reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_09024 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces Payandeh, Amirreza Song, Daeun Nazeri, Mohammad Liang, Jing Mukherjee, Praneel Raj, Amir Hossain Kong, Yangzhe Manocha, Dinesh Xiao, Xuesu Computer Vision and Pattern Recognition Human-Computer Interaction Robotics Most existing social robot navigation techniques either leverage hand-crafted rules or human demonstrations to connect robot perception to socially compliant actions. However, there remains a significant gap in effectively translating perception into socially compliant actions, much like how human reasoning naturally occurs in dynamic environments. Considering the recent success of Vision-Language Models (VLMs), we propose using language to bridge the gap in human-like reasoning between perception and socially aware robot actions. We create a vision-language dataset, Social robot Navigation via Explainable Interactions (SNEI), featuring 40K human-annotated Visual Question Answers (VQAs) based on 2K human-robot social interactions in unstructured, crowded public spaces, spanning perception, prediction, chain-of-thought reasoning, action, and explanation. We fine-tune a VLM, Social-LLaVA, using SNEI to demonstrate the practical application of our dataset. Social-LLaVA outperforms state-of-the-art models like GPT-4V and Gemini, based on the average of fifteen different human-judge scores across 50 VQA. Deployed onboard a mobile robot, Social-LLaVA enables human-like reasoning, marking a promising step toward socially compliant robot navigation in dynamic public spaces through language reasoning. |
| title | Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces |
| topic | Computer Vision and Pattern Recognition Human-Computer Interaction Robotics |
| url | https://arxiv.org/abs/2501.09024 |