Hearing Hands: Generating Sounds from Physical Interactions in 3D Scenes
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913889435779072 |
|---|---|
| author | Dou, Yiming Oh, Wonseok Luo, Yuqing Loquercio, Antonio Owens, Andrew |
| author_facet | Dou, Yiming Oh, Wonseok Luo, Yuqing Loquercio, Antonio Owens, Andrew |
| contents | We study the problem of making 3D scene reconstructions interactive by asking the following question: can we predict the sounds of human hands physically interacting with a scene? First, we record a video of a human manipulating objects within a 3D scene using their hands. We then use these action-sound pairs to train a rectified flow model to map 3D hand trajectories to their corresponding audio. At test time, a user can query the model for other actions, parameterized as sequences of hand poses, to estimate their corresponding sounds. In our experiments, we find that our generated sounds accurately convey material properties and actions, and that they are often indistinguishable to human observers from real sounds. Project page: https://www.yimingdou.com/hearing_hands/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_09989 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Hearing Hands: Generating Sounds from Physical Interactions in 3D Scenes Dou, Yiming Oh, Wonseok Luo, Yuqing Loquercio, Antonio Owens, Andrew Computer Vision and Pattern Recognition We study the problem of making 3D scene reconstructions interactive by asking the following question: can we predict the sounds of human hands physically interacting with a scene? First, we record a video of a human manipulating objects within a 3D scene using their hands. We then use these action-sound pairs to train a rectified flow model to map 3D hand trajectories to their corresponding audio. At test time, a user can query the model for other actions, parameterized as sequences of hand poses, to estimate their corresponding sounds. In our experiments, we find that our generated sounds accurately convey material properties and actions, and that they are often indistinguishable to human observers from real sounds. Project page: https://www.yimingdou.com/hearing_hands/ |
| title | Hearing Hands: Generating Sounds from Physical Interactions in 3D Scenes |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2506.09989 |