HARMONI: Multimodal Personalization of Multi-User Human-Robot Interactions with LLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910002518687744 |
|---|---|
| author | Malécot, Jeanne Rahimi, Hamed Cattoni, Jeanne Samson, Marie Abrini, Mouad Khoramshahi, Mahdi Pino, Maribel Chetouani, Mohamed |
| author_facet | Malécot, Jeanne Rahimi, Hamed Cattoni, Jeanne Samson, Marie Abrini, Mouad Khoramshahi, Mahdi Pino, Maribel Chetouani, Mohamed |
| contents | Existing human-robot interaction systems often lack mechanisms for sustained personalization and dynamic adaptation in multi-user environments, limiting their effectiveness in real-world deployments. We present HARMONI, a multimodal personalization framework that leverages large language models to enable socially assistive robots to manage long-term multi-user interactions. The framework integrates four key modules: (i) a perception module that identifies active speakers and extracts multimodal input; (ii) a world modeling module that maintains representations of the environment and short-term conversational context; (iii) a user modeling module that updates long-term speaker-specific profiles; and (iv) a generation module that produces contextually grounded and ethically informed responses. Through extensive evaluation and ablation studies on four datasets, as well as a real-world scenario-driven user-study in a nursing home environment, we demonstrate that HARMONI supports robust speaker identification, online memory updating, and ethically aligned personalization, outperforming baseline LLM-driven approaches in user modeling accuracy, personalization quality, and user satisfaction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_19839 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | HARMONI: Multimodal Personalization of Multi-User Human-Robot Interactions with LLMs Malécot, Jeanne Rahimi, Hamed Cattoni, Jeanne Samson, Marie Abrini, Mouad Khoramshahi, Mahdi Pino, Maribel Chetouani, Mohamed Robotics Artificial Intelligence Human-Computer Interaction Existing human-robot interaction systems often lack mechanisms for sustained personalization and dynamic adaptation in multi-user environments, limiting their effectiveness in real-world deployments. We present HARMONI, a multimodal personalization framework that leverages large language models to enable socially assistive robots to manage long-term multi-user interactions. The framework integrates four key modules: (i) a perception module that identifies active speakers and extracts multimodal input; (ii) a world modeling module that maintains representations of the environment and short-term conversational context; (iii) a user modeling module that updates long-term speaker-specific profiles; and (iv) a generation module that produces contextually grounded and ethically informed responses. Through extensive evaluation and ablation studies on four datasets, as well as a real-world scenario-driven user-study in a nursing home environment, we demonstrate that HARMONI supports robust speaker identification, online memory updating, and ethically aligned personalization, outperforming baseline LLM-driven approaches in user modeling accuracy, personalization quality, and user satisfaction. |
| title | HARMONI: Multimodal Personalization of Multi-User Human-Robot Interactions with LLMs |
| topic | Robotics Artificial Intelligence Human-Computer Interaction |
| url | https://arxiv.org/abs/2601.19839 |