The Sound of Simulation: Learning Multimodal Sim-to-Real Robot Policies with Generative Audio
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911167436292096 |
|---|---|
| author | Wang, Renhao Geng, Haoran Li, Tingle Wang, Feishi Anumanchipalli, Gopala Darrell, Trevor Li, Boyi Abbeel, Pieter Malik, Jitendra Efros, Alexei A. |
| author_facet | Wang, Renhao Geng, Haoran Li, Tingle Wang, Feishi Anumanchipalli, Gopala Darrell, Trevor Li, Boyi Abbeel, Pieter Malik, Jitendra Efros, Alexei A. |
| contents | Robots must integrate multiple sensory modalities to act effectively in the real world. Yet, learning such multimodal policies at scale remains challenging. Simulation offers a viable solution, but while vision has benefited from high-fidelity simulators, other modalities (e.g. sound) can be notoriously difficult to simulate. As a result, sim-to-real transfer has succeeded primarily in vision-based tasks, with multimodal transfer still largely unrealized. In this work, we tackle these challenges by introducing MultiGen, a framework that integrates large-scale generative models into traditional physics simulators, enabling multisensory simulation. We showcase our framework on the dynamic task of robot pouring, which inherently relies on multimodal feedback. By synthesizing realistic audio conditioned on simulation video, our method enables training on rich audiovisual trajectories -- without any real robot data. We demonstrate effective zero-shot transfer to real-world pouring with novel containers and liquids, highlighting the potential of generative modeling to both simulate hard-to-model modalities and close the multimodal sim-to-real gap. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_02864 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The Sound of Simulation: Learning Multimodal Sim-to-Real Robot Policies with Generative Audio Wang, Renhao Geng, Haoran Li, Tingle Wang, Feishi Anumanchipalli, Gopala Darrell, Trevor Li, Boyi Abbeel, Pieter Malik, Jitendra Efros, Alexei A. Robotics Computer Vision and Pattern Recognition Robots must integrate multiple sensory modalities to act effectively in the real world. Yet, learning such multimodal policies at scale remains challenging. Simulation offers a viable solution, but while vision has benefited from high-fidelity simulators, other modalities (e.g. sound) can be notoriously difficult to simulate. As a result, sim-to-real transfer has succeeded primarily in vision-based tasks, with multimodal transfer still largely unrealized. In this work, we tackle these challenges by introducing MultiGen, a framework that integrates large-scale generative models into traditional physics simulators, enabling multisensory simulation. We showcase our framework on the dynamic task of robot pouring, which inherently relies on multimodal feedback. By synthesizing realistic audio conditioned on simulation video, our method enables training on rich audiovisual trajectories -- without any real robot data. We demonstrate effective zero-shot transfer to real-world pouring with novel containers and liquids, highlighting the potential of generative modeling to both simulate hard-to-model modalities and close the multimodal sim-to-real gap. |
| title | The Sound of Simulation: Learning Multimodal Sim-to-Real Robot Policies with Generative Audio |
| topic | Robotics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2507.02864 |