Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911972970201088 |
|---|---|
| author | Yuan, Ying Che, Haichuan Qin, Yuzhe Huang, Binghao Yin, Zhao-Heng Lee, Kang-Won Wu, Yi Lim, Soo-Chul Wang, Xiaolong |
| author_facet | Yuan, Ying Che, Haichuan Qin, Yuzhe Huang, Binghao Yin, Zhao-Heng Lee, Kang-Won Wu, Yi Lim, Soo-Chul Wang, Xiaolong |
| contents | Executing contact-rich manipulation tasks necessitates the fusion of tactile and visual feedback. However, the distinct nature of these modalities poses significant challenges. In this paper, we introduce a system that leverages visual and tactile sensory inputs to enable dexterous in-hand manipulation. Specifically, we propose Robot Synesthesia, a novel point cloud-based tactile representation inspired by human tactile-visual synesthesia. This approach allows for the simultaneous and seamless integration of both sensory inputs, offering richer spatial information and facilitating better reasoning about robot actions. The method, trained in a simulated environment and then deployed to a real robot, is applicable to various in-hand object rotation tasks. Comprehensive ablations are performed on how the integration of vision and touch can improve reinforcement learning and Sim2Real performance. Our project page is available at https://yingyuan0414.github.io/visuotactile/ . |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_01853 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing Yuan, Ying Che, Haichuan Qin, Yuzhe Huang, Binghao Yin, Zhao-Heng Lee, Kang-Won Wu, Yi Lim, Soo-Chul Wang, Xiaolong Robotics Computer Vision and Pattern Recognition Machine Learning Executing contact-rich manipulation tasks necessitates the fusion of tactile and visual feedback. However, the distinct nature of these modalities poses significant challenges. In this paper, we introduce a system that leverages visual and tactile sensory inputs to enable dexterous in-hand manipulation. Specifically, we propose Robot Synesthesia, a novel point cloud-based tactile representation inspired by human tactile-visual synesthesia. This approach allows for the simultaneous and seamless integration of both sensory inputs, offering richer spatial information and facilitating better reasoning about robot actions. The method, trained in a simulated environment and then deployed to a real robot, is applicable to various in-hand object rotation tasks. Comprehensive ablations are performed on how the integration of vision and touch can improve reinforcement learning and Sim2Real performance. Our project page is available at https://yingyuan0414.github.io/visuotactile/ . |
| title | Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing |
| topic | Robotics Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2312.01853 |