Tactile Beyond Pixels: Multisensory Touch Representations for Robot Manipulation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909651606437888 |
|---|---|
| author | Higuera, Carolina Sharma, Akash Fan, Taosha Bodduluri, Chaithanya Krishna Boots, Byron Kaess, Michael Lambeta, Mike Wu, Tingfan Liu, Zixi Hogan, Francois Robert Mukadam, Mustafa |
| author_facet | Higuera, Carolina Sharma, Akash Fan, Taosha Bodduluri, Chaithanya Krishna Boots, Byron Kaess, Michael Lambeta, Mike Wu, Tingfan Liu, Zixi Hogan, Francois Robert Mukadam, Mustafa |
| contents | We present Sparsh-X, the first multisensory touch representations across four tactile modalities: image, audio, motion, and pressure. Trained on ~1M contact-rich interactions collected with the Digit 360 sensor, Sparsh-X captures complementary touch signals at diverse temporal and spatial scales. By leveraging self-supervised learning, Sparsh-X fuses these modalities into a unified representation that captures physical properties useful for robot manipulation tasks. We study how to effectively integrate real-world touch representations for both imitation learning and tactile adaptation of sim-trained policies, showing that Sparsh-X boosts policy success rates by 63% over an end-to-end model using tactile images and improves robustness by 90% in recovering object states from touch. Finally, we benchmark Sparsh-X ability to make inferences about physical properties, such as object-action identification, material-quantity estimation, and force estimation. Sparsh-X improves accuracy in characterizing physical properties by 48% compared to end-to-end approaches, demonstrating the advantages of multisensory pretraining for capturing features essential for dexterous manipulation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_14754 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Tactile Beyond Pixels: Multisensory Touch Representations for Robot Manipulation Higuera, Carolina Sharma, Akash Fan, Taosha Bodduluri, Chaithanya Krishna Boots, Byron Kaess, Michael Lambeta, Mike Wu, Tingfan Liu, Zixi Hogan, Francois Robert Mukadam, Mustafa Robotics We present Sparsh-X, the first multisensory touch representations across four tactile modalities: image, audio, motion, and pressure. Trained on ~1M contact-rich interactions collected with the Digit 360 sensor, Sparsh-X captures complementary touch signals at diverse temporal and spatial scales. By leveraging self-supervised learning, Sparsh-X fuses these modalities into a unified representation that captures physical properties useful for robot manipulation tasks. We study how to effectively integrate real-world touch representations for both imitation learning and tactile adaptation of sim-trained policies, showing that Sparsh-X boosts policy success rates by 63% over an end-to-end model using tactile images and improves robustness by 90% in recovering object states from touch. Finally, we benchmark Sparsh-X ability to make inferences about physical properties, such as object-action identification, material-quantity estimation, and force estimation. Sparsh-X improves accuracy in characterizing physical properties by 48% compared to end-to-end approaches, demonstrating the advantages of multisensory pretraining for capturing features essential for dexterous manipulation. |
| title | Tactile Beyond Pixels: Multisensory Touch Representations for Robot Manipulation |
| topic | Robotics |
| url | https://arxiv.org/abs/2506.14754 |