Chain-of-Modality: Learning Manipulation Programs from Multimodal Human Videos with Vision-Language-Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910913947238400 |
|---|---|
| author | Wang, Chen Xia, Fei Yu, Wenhao Zhang, Tingnan Zhang, Ruohan Liu, C. Karen Fei-Fei, Li Tan, Jie Liang, Jacky |
| author_facet | Wang, Chen Xia, Fei Yu, Wenhao Zhang, Tingnan Zhang, Ruohan Liu, C. Karen Fei-Fei, Li Tan, Jie Liang, Jacky |
| contents | Learning to perform manipulation tasks from human videos is a promising approach for teaching robots. However, many manipulation tasks require changing control parameters during task execution, such as force, which visual data alone cannot capture. In this work, we leverage sensing devices such as armbands that measure human muscle activities and microphones that record sound, to capture the details in the human manipulation process, and enable robots to extract task plans and control parameters to perform the same task. To achieve this, we introduce Chain-of-Modality (CoM), a prompting strategy that enables Vision Language Models to reason about multimodal human demonstration data -- videos coupled with muscle or audio signals. By progressively integrating information from each modality, CoM refines a task plan and generates detailed control parameters, enabling robots to perform manipulation tasks based on a single multimodal human video prompt. Our experiments show that CoM delivers a threefold improvement in accuracy for extracting task plans and control parameters compared to baselines, with strong generalization to new task setups and objects in real-world robot experiments. Videos and code are available at https://chain-of-modality.github.io |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_13351 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Chain-of-Modality: Learning Manipulation Programs from Multimodal Human Videos with Vision-Language-Models Wang, Chen Xia, Fei Yu, Wenhao Zhang, Tingnan Zhang, Ruohan Liu, C. Karen Fei-Fei, Li Tan, Jie Liang, Jacky Robotics Artificial Intelligence Human-Computer Interaction Machine Learning Multimedia Learning to perform manipulation tasks from human videos is a promising approach for teaching robots. However, many manipulation tasks require changing control parameters during task execution, such as force, which visual data alone cannot capture. In this work, we leverage sensing devices such as armbands that measure human muscle activities and microphones that record sound, to capture the details in the human manipulation process, and enable robots to extract task plans and control parameters to perform the same task. To achieve this, we introduce Chain-of-Modality (CoM), a prompting strategy that enables Vision Language Models to reason about multimodal human demonstration data -- videos coupled with muscle or audio signals. By progressively integrating information from each modality, CoM refines a task plan and generates detailed control parameters, enabling robots to perform manipulation tasks based on a single multimodal human video prompt. Our experiments show that CoM delivers a threefold improvement in accuracy for extracting task plans and control parameters compared to baselines, with strong generalization to new task setups and objects in real-world robot experiments. Videos and code are available at https://chain-of-modality.github.io |
| title | Chain-of-Modality: Learning Manipulation Programs from Multimodal Human Videos with Vision-Language-Models |
| topic | Robotics Artificial Intelligence Human-Computer Interaction Machine Learning Multimedia |
| url | https://arxiv.org/abs/2504.13351 |