USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactions
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916634821656576 |
|---|---|
| author | Rahimi, Hamed Bahaj, Adil Abrini, Mouad Khoramshahi, Mahdi Ghogho, Mounir Chetouani, Mohamed |
| author_facet | Rahimi, Hamed Bahaj, Adil Abrini, Mouad Khoramshahi, Mahdi Ghogho, Mounir Chetouani, Mohamed |
| contents | The integration of vision-language models into robotic systems constitutes a significant advancement in enabling machines to interact with their surroundings in a more intuitive manner. While VLMs offer rich multimodal reasoning, existing approaches lack user-specific adaptability, often relying on generic interaction paradigms that fail to account for individual behavioral, contextual, or socio-emotional nuances. When customization is attempted, ethical concerns arise from unmitigated biases in user data, risking exclusion or unfair treatment. To address these dual challenges, we propose User-VLM 360°, a holistic framework integrating multimodal user modeling with bias-aware optimization. Our approach features: (1) user-aware tuning that adapts interactions in real time using visual-linguistic signals; (2) bias mitigation via preference optimization; and (3) curated 360° socio-emotive interaction datasets annotated with demographic, emotion, and relational metadata. Evaluations across eight benchmarks demonstrate state-of-the-art results: +35.3% F1 in personalized VQA, +47.5% F1 in facial features understanding, 15% bias reduction, and 30X speedup over baselines. Ablation studies confirm component efficacy, and deployment on the Pepper robot validates real-time adaptability across diverse users. We open-source parameter-efficient 3B/10B models and an ethical verification framework for responsible adaptation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_10636 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactions Rahimi, Hamed Bahaj, Adil Abrini, Mouad Khoramshahi, Mahdi Ghogho, Mounir Chetouani, Mohamed Artificial Intelligence Human-Computer Interaction Robotics The integration of vision-language models into robotic systems constitutes a significant advancement in enabling machines to interact with their surroundings in a more intuitive manner. While VLMs offer rich multimodal reasoning, existing approaches lack user-specific adaptability, often relying on generic interaction paradigms that fail to account for individual behavioral, contextual, or socio-emotional nuances. When customization is attempted, ethical concerns arise from unmitigated biases in user data, risking exclusion or unfair treatment. To address these dual challenges, we propose User-VLM 360°, a holistic framework integrating multimodal user modeling with bias-aware optimization. Our approach features: (1) user-aware tuning that adapts interactions in real time using visual-linguistic signals; (2) bias mitigation via preference optimization; and (3) curated 360° socio-emotive interaction datasets annotated with demographic, emotion, and relational metadata. Evaluations across eight benchmarks demonstrate state-of-the-art results: +35.3% F1 in personalized VQA, +47.5% F1 in facial features understanding, 15% bias reduction, and 30X speedup over baselines. Ablation studies confirm component efficacy, and deployment on the Pepper robot validates real-time adaptability across diverse users. We open-source parameter-efficient 3B/10B models and an ethical verification framework for responsible adaptation. |
| title | USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactions |
| topic | Artificial Intelligence Human-Computer Interaction Robotics |
| url | https://arxiv.org/abs/2502.10636 |