RoboOmni: Proactive Robot Manipulation in Omni-modal Context
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908624296607744 |
|---|---|
| author | Wang, Siyin Fu, Jinlan Liu, Feihong He, Xinzhe Wu, Huangxuan Shi, Junhao Huang, Kexin Fei, Zhaoye Gong, Jingjing Wu, Zuxuan Jiang, Yu-Gang Ng, See-Kiong Chua, Tat-Seng Qiu, Xipeng |
| author_facet | Wang, Siyin Fu, Jinlan Liu, Feihong He, Xinzhe Wu, Huangxuan Shi, Junhao Huang, Kexin Fei, Zhaoye Gong, Jingjing Wu, Zuxuan Jiang, Yu-Gang Ng, See-Kiong Chua, Tat-Seng Qiu, Xipeng |
| contents | Recent advances in Multimodal Large Language Models (MLLMs) have driven rapid progress in Vision-Language-Action (VLA) models for robotic manipulation. Although effective in many scenarios, current approaches largely rely on explicit instructions, whereas in real-world interactions, humans rarely issue instructions directly. Effective collaboration requires robots to infer user intentions proactively. In this work, we introduce cross-modal contextual instructions, a new setting where intent is derived from spoken dialogue, environmental sounds, and visual cues rather than explicit commands. To address this new setting, we present RoboOmni, a Perceiver-Thinker-Talker-Executor framework based on end-to-end omni-modal LLMs that unifies intention recognition, interaction confirmation, and action execution. RoboOmni fuses auditory and visual signals spatiotemporally for robust intention recognition, while supporting direct speech interaction. To address the absence of training data for proactive intention recognition in robotic manipulation, we build OmniAction, comprising 140k episodes, 5k+ speakers, 2.4k event sounds, 640 backgrounds, and six contextual instruction types. Experiments in simulation and real-world settings show that RoboOmni surpasses text- and ASR-based baselines in success rate, inference speed, intention recognition, and proactive assistance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_23763 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RoboOmni: Proactive Robot Manipulation in Omni-modal Context Wang, Siyin Fu, Jinlan Liu, Feihong He, Xinzhe Wu, Huangxuan Shi, Junhao Huang, Kexin Fei, Zhaoye Gong, Jingjing Wu, Zuxuan Jiang, Yu-Gang Ng, See-Kiong Chua, Tat-Seng Qiu, Xipeng Robotics Computation and Language Computer Vision and Pattern Recognition Recent advances in Multimodal Large Language Models (MLLMs) have driven rapid progress in Vision-Language-Action (VLA) models for robotic manipulation. Although effective in many scenarios, current approaches largely rely on explicit instructions, whereas in real-world interactions, humans rarely issue instructions directly. Effective collaboration requires robots to infer user intentions proactively. In this work, we introduce cross-modal contextual instructions, a new setting where intent is derived from spoken dialogue, environmental sounds, and visual cues rather than explicit commands. To address this new setting, we present RoboOmni, a Perceiver-Thinker-Talker-Executor framework based on end-to-end omni-modal LLMs that unifies intention recognition, interaction confirmation, and action execution. RoboOmni fuses auditory and visual signals spatiotemporally for robust intention recognition, while supporting direct speech interaction. To address the absence of training data for proactive intention recognition in robotic manipulation, we build OmniAction, comprising 140k episodes, 5k+ speakers, 2.4k event sounds, 640 backgrounds, and six contextual instruction types. Experiments in simulation and real-world settings show that RoboOmni surpasses text- and ASR-based baselines in success rate, inference speed, intention recognition, and proactive assistance. |
| title | RoboOmni: Proactive Robot Manipulation in Omni-modal Context |
| topic | Robotics Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2510.23763 |