Hearing Anywhere in Any Environment
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912413953032192 |
|---|---|
| author | Liu, Xiulong Kumar, Anurag Calamia, Paul Amengual, Sebastia V. Murdock, Calvin Ananthabhotla, Ishwarya Robinson, Philip Shlizerman, Eli Ithapu, Vamsi Krishna Gao, Ruohan |
| author_facet | Liu, Xiulong Kumar, Anurag Calamia, Paul Amengual, Sebastia V. Murdock, Calvin Ananthabhotla, Ishwarya Robinson, Philip Shlizerman, Eli Ithapu, Vamsi Krishna Gao, Ruohan |
| contents | In mixed reality applications, a realistic acoustic experience in spatial environments is as crucial as the visual experience for achieving true immersion. Despite recent advances in neural approaches for Room Impulse Response (RIR) estimation, most existing methods are limited to the single environment on which they are trained, lacking the ability to generalize to new rooms with different geometries and surface materials. We aim to develop a unified model capable of reconstructing the spatial acoustic experience of any environment with minimum additional measurements. To this end, we present xRIR, a framework for cross-room RIR prediction. The core of our generalizable approach lies in combining a geometric feature extractor, which captures spatial context from panorama depth images, with a RIR encoder that extracts detailed acoustic features from only a few reference RIR samples. To evaluate our method, we introduce ACOUSTICROOMS, a new dataset featuring high-fidelity simulation of over 300,000 RIRs from 260 rooms. Experiments show that our method strongly outperforms a series of baselines. Furthermore, we successfully perform sim-to-real transfer by evaluating our model on four real-world environments, demonstrating the generalizability of our approach and the realism of our dataset. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_10746 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Hearing Anywhere in Any Environment Liu, Xiulong Kumar, Anurag Calamia, Paul Amengual, Sebastia V. Murdock, Calvin Ananthabhotla, Ishwarya Robinson, Philip Shlizerman, Eli Ithapu, Vamsi Krishna Gao, Ruohan Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Sound Audio and Speech Processing In mixed reality applications, a realistic acoustic experience in spatial environments is as crucial as the visual experience for achieving true immersion. Despite recent advances in neural approaches for Room Impulse Response (RIR) estimation, most existing methods are limited to the single environment on which they are trained, lacking the ability to generalize to new rooms with different geometries and surface materials. We aim to develop a unified model capable of reconstructing the spatial acoustic experience of any environment with minimum additional measurements. To this end, we present xRIR, a framework for cross-room RIR prediction. The core of our generalizable approach lies in combining a geometric feature extractor, which captures spatial context from panorama depth images, with a RIR encoder that extracts detailed acoustic features from only a few reference RIR samples. To evaluate our method, we introduce ACOUSTICROOMS, a new dataset featuring high-fidelity simulation of over 300,000 RIRs from 260 rooms. Experiments show that our method strongly outperforms a series of baselines. Furthermore, we successfully perform sim-to-real transfer by evaluating our model on four real-world environments, demonstrating the generalizability of our approach and the realism of our dataset. |
| title | Hearing Anywhere in Any Environment |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2504.10746 |