Disentangled Acoustic Fields For Multimodal Physical Scene Understanding
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911956753973248 |
|---|---|
| author | Yin, Jie Luo, Andrew Du, Yilun Cherian, Anoop Marks, Tim K. Roux, Jonathan Le Gan, Chuang |
| author_facet | Yin, Jie Luo, Andrew Du, Yilun Cherian, Anoop Marks, Tim K. Roux, Jonathan Le Gan, Chuang |
| contents | We study the problem of multimodal physical scene understanding, where an embodied agent needs to find fallen objects by inferring object properties, direction, and distance of an impact sound source. Previous works adopt feed-forward neural networks to directly regress the variables from sound, leading to poor generalization and domain adaptation issues. In this paper, we illustrate that learning a disentangled model of acoustic formation, referred to as disentangled acoustic field (DAF), to capture the sound generation and propagation process, enables the embodied agent to construct a spatial uncertainty map over where the objects may have fallen. We demonstrate that our analysis-by-synthesis framework can jointly infer sound properties by explicitly decomposing and factorizing the latent space of the disentangled model. We further show that the spatial uncertainty map can significantly improve the success rate for the localization of fallen objects by proposing multiple plausible exploration locations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_11333 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Disentangled Acoustic Fields For Multimodal Physical Scene Understanding Yin, Jie Luo, Andrew Du, Yilun Cherian, Anoop Marks, Tim K. Roux, Jonathan Le Gan, Chuang Robotics Sound Audio and Speech Processing We study the problem of multimodal physical scene understanding, where an embodied agent needs to find fallen objects by inferring object properties, direction, and distance of an impact sound source. Previous works adopt feed-forward neural networks to directly regress the variables from sound, leading to poor generalization and domain adaptation issues. In this paper, we illustrate that learning a disentangled model of acoustic formation, referred to as disentangled acoustic field (DAF), to capture the sound generation and propagation process, enables the embodied agent to construct a spatial uncertainty map over where the objects may have fallen. We demonstrate that our analysis-by-synthesis framework can jointly infer sound properties by explicitly decomposing and factorizing the latent space of the disentangled model. We further show that the spatial uncertainty map can significantly improve the success rate for the localization of fallen objects by proposing multiple plausible exploration locations. |
| title | Disentangled Acoustic Fields For Multimodal Physical Scene Understanding |
| topic | Robotics Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2407.11333 |