DiffSound: Differentiable Modal Sound Rendering and Inverse Rendering for Diverse Inference Tasks
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917780301807616 |
|---|---|
| author | Jin, Xutong Xu, Chenxi Gao, Ruohan Wu, Jiajun Wang, Guoping Li, Sheng |
| author_facet | Jin, Xutong Xu, Chenxi Gao, Ruohan Wu, Jiajun Wang, Guoping Li, Sheng |
| contents | Accurately estimating and simulating the physical properties of objects from real-world sound recordings is of great practical importance in the fields of vision, graphics, and robotics. However, the progress in these directions has been limited -- prior differentiable rigid or soft body simulation techniques cannot be directly applied to modal sound synthesis due to the high sampling rate of audio, while previous audio synthesizers often do not fully model the accurate physical properties of the sounding objects. We propose DiffSound, a differentiable sound rendering framework for physics-based modal sound synthesis, which is based on an implicit shape representation, a new high-order finite element analysis module, and a differentiable audio synthesizer. Our framework can solve a wide range of inverse problems thanks to the differentiability of the entire pipeline, including physical parameter estimation, geometric shape reasoning, and impact position prediction. Experimental results demonstrate the effectiveness of our approach, highlighting its ability to accurately reproduce the target sound in a physics-based manner. DiffSound serves as a valuable tool for various sound synthesis and analysis applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_13486 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DiffSound: Differentiable Modal Sound Rendering and Inverse Rendering for Diverse Inference Tasks Jin, Xutong Xu, Chenxi Gao, Ruohan Wu, Jiajun Wang, Guoping Li, Sheng Sound Audio and Speech Processing Accurately estimating and simulating the physical properties of objects from real-world sound recordings is of great practical importance in the fields of vision, graphics, and robotics. However, the progress in these directions has been limited -- prior differentiable rigid or soft body simulation techniques cannot be directly applied to modal sound synthesis due to the high sampling rate of audio, while previous audio synthesizers often do not fully model the accurate physical properties of the sounding objects. We propose DiffSound, a differentiable sound rendering framework for physics-based modal sound synthesis, which is based on an implicit shape representation, a new high-order finite element analysis module, and a differentiable audio synthesizer. Our framework can solve a wide range of inverse problems thanks to the differentiability of the entire pipeline, including physical parameter estimation, geometric shape reasoning, and impact position prediction. Experimental results demonstrate the effectiveness of our approach, highlighting its ability to accurately reproduce the target sound in a physics-based manner. DiffSound serves as a valuable tool for various sound synthesis and analysis applications. |
| title | DiffSound: Differentiable Modal Sound Rendering and Inverse Rendering for Diverse Inference Tasks |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2409.13486 |