ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910681147637760 |
|---|---|
| author | Liu, Zeyi Chi, Cheng Cousineau, Eric Kuppuswamy, Naveen Burchfiel, Benjamin Song, Shuran |
| author_facet | Liu, Zeyi Chi, Cheng Cousineau, Eric Kuppuswamy, Naveen Burchfiel, Benjamin Song, Shuran |
| contents | Audio signals provide rich information for the robot interaction and object properties through contact. This information can surprisingly ease the learning of contact-rich robot manipulation skills, especially when the visual information alone is ambiguous or incomplete. However, the usage of audio data in robot manipulation has been constrained to teleoperated demonstrations collected by either attaching a microphone to the robot or object, which significantly limits its usage in robot learning pipelines. In this work, we introduce ManiWAV: an 'ear-in-hand' data collection device to collect in-the-wild human demonstrations with synchronous audio and visual feedback, and a corresponding policy interface to learn robot manipulation policy directly from the demonstrations. We demonstrate the capabilities of our system through four contact-rich manipulation tasks that require either passively sensing the contact events and modes, or actively sensing the object surface materials and states. In addition, we show that our system can generalize to unseen in-the-wild environments by learning from diverse in-the-wild human demonstrations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_19464 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data Liu, Zeyi Chi, Cheng Cousineau, Eric Kuppuswamy, Naveen Burchfiel, Benjamin Song, Shuran Robotics Artificial Intelligence Computer Vision and Pattern Recognition Sound Audio and Speech Processing Audio signals provide rich information for the robot interaction and object properties through contact. This information can surprisingly ease the learning of contact-rich robot manipulation skills, especially when the visual information alone is ambiguous or incomplete. However, the usage of audio data in robot manipulation has been constrained to teleoperated demonstrations collected by either attaching a microphone to the robot or object, which significantly limits its usage in robot learning pipelines. In this work, we introduce ManiWAV: an 'ear-in-hand' data collection device to collect in-the-wild human demonstrations with synchronous audio and visual feedback, and a corresponding policy interface to learn robot manipulation policy directly from the demonstrations. We demonstrate the capabilities of our system through four contact-rich manipulation tasks that require either passively sensing the contact events and modes, or actively sensing the object surface materials and states. In addition, we show that our system can generalize to unseen in-the-wild environments by learning from diverse in-the-wild human demonstrations. |
| title | ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data |
| topic | Robotics Artificial Intelligence Computer Vision and Pattern Recognition Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2406.19464 |