SoundVista: Novel-View Ambient Sound Synthesis via Visual-Acoustic Binding
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912315891253248 |
|---|---|
| author | Chen, Mingfei Gebru, Israel D. Ananthabhotla, Ishwarya Richardt, Christian Markovic, Dejan Sandakly, Jake Krenn, Steven Keebler, Todd Shlizerman, Eli Richard, Alexander |
| author_facet | Chen, Mingfei Gebru, Israel D. Ananthabhotla, Ishwarya Richardt, Christian Markovic, Dejan Sandakly, Jake Krenn, Steven Keebler, Todd Shlizerman, Eli Richard, Alexander |
| contents | We introduce SoundVista, a method to generate the ambient sound of an arbitrary scene at novel viewpoints. Given a pre-acquired recording of the scene from sparsely distributed microphones, SoundVista can synthesize the sound of that scene from an unseen target viewpoint. The method learns the underlying acoustic transfer function that relates the signals acquired at the distributed microphones to the signal at the target viewpoint, using a limited number of known recordings. Unlike existing works, our method does not require constraints or prior knowledge of sound source details. Moreover, our method efficiently adapts to diverse room layouts, reference microphone configurations and unseen environments. To enable this, we introduce a visual-acoustic binding module that learns visual embeddings linked with local acoustic properties from panoramic RGB and depth data. We first leverage these embeddings to optimize the placement of reference microphones in any given scene. During synthesis, we leverage multiple embeddings extracted from reference locations to get adaptive weights for their contribution, conditioned on target viewpoint. We benchmark the task on both publicly available data and real-world settings. We demonstrate significant improvements over existing methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_05576 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SoundVista: Novel-View Ambient Sound Synthesis via Visual-Acoustic Binding Chen, Mingfei Gebru, Israel D. Ananthabhotla, Ishwarya Richardt, Christian Markovic, Dejan Sandakly, Jake Krenn, Steven Keebler, Todd Shlizerman, Eli Richard, Alexander Sound Artificial Intelligence Computer Vision and Pattern Recognition Multimedia We introduce SoundVista, a method to generate the ambient sound of an arbitrary scene at novel viewpoints. Given a pre-acquired recording of the scene from sparsely distributed microphones, SoundVista can synthesize the sound of that scene from an unseen target viewpoint. The method learns the underlying acoustic transfer function that relates the signals acquired at the distributed microphones to the signal at the target viewpoint, using a limited number of known recordings. Unlike existing works, our method does not require constraints or prior knowledge of sound source details. Moreover, our method efficiently adapts to diverse room layouts, reference microphone configurations and unseen environments. To enable this, we introduce a visual-acoustic binding module that learns visual embeddings linked with local acoustic properties from panoramic RGB and depth data. We first leverage these embeddings to optimize the placement of reference microphones in any given scene. During synthesis, we leverage multiple embeddings extracted from reference locations to get adaptive weights for their contribution, conditioned on target viewpoint. We benchmark the task on both publicly available data and real-world settings. We demonstrate significant improvements over existing methods. |
| title | SoundVista: Novel-View Ambient Sound Synthesis via Visual-Acoustic Binding |
| topic | Sound Artificial Intelligence Computer Vision and Pattern Recognition Multimedia |
| url | https://arxiv.org/abs/2504.05576 |