Zero-Shot Mono-to-Binaural Speech Synthesis
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910971915665408 |
|---|---|
| author | Levkovitch, Alon Salazar, Julian Mariooryad, Soroosh Skerry-Ryan, RJ Bar, Nadav Kleijn, Bastiaan Nachmani, Eliya |
| author_facet | Levkovitch, Alon Salazar, Julian Mariooryad, Soroosh Skerry-Ryan, RJ Bar, Nadav Kleijn, Bastiaan Nachmani, Eliya |
| contents | We present ZeroBAS, a neural method to synthesize binaural audio from monaural audio recordings and positional information without training on any binaural data. To our knowledge, this is the first published zero-shot neural approach to mono-to-binaural audio synthesis. Specifically, we show that a parameter-free geometric time warping and amplitude scaling based on source location suffices to get an initial binaural synthesis that can be refined by iteratively applying a pretrained denoising vocoder. Furthermore, we find this leads to generalization across room conditions, which we measure by introducing a new dataset, TUT Mono-to-Binaural, to evaluate state-of-the-art monaural-to-binaural synthesis methods on unseen conditions. Our zero-shot method is perceptually on-par with the performance of supervised methods on the standard mono-to-binaural dataset, and even surpasses them on our out-of-distribution TUT Mono-to-Binaural dataset. Our results highlight the potential of pretrained generative audio models and zero-shot learning to unlock robust binaural audio synthesis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_08356 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Zero-Shot Mono-to-Binaural Speech Synthesis Levkovitch, Alon Salazar, Julian Mariooryad, Soroosh Skerry-Ryan, RJ Bar, Nadav Kleijn, Bastiaan Nachmani, Eliya Sound Machine Learning Audio and Speech Processing We present ZeroBAS, a neural method to synthesize binaural audio from monaural audio recordings and positional information without training on any binaural data. To our knowledge, this is the first published zero-shot neural approach to mono-to-binaural audio synthesis. Specifically, we show that a parameter-free geometric time warping and amplitude scaling based on source location suffices to get an initial binaural synthesis that can be refined by iteratively applying a pretrained denoising vocoder. Furthermore, we find this leads to generalization across room conditions, which we measure by introducing a new dataset, TUT Mono-to-Binaural, to evaluate state-of-the-art monaural-to-binaural synthesis methods on unseen conditions. Our zero-shot method is perceptually on-par with the performance of supervised methods on the standard mono-to-binaural dataset, and even surpasses them on our out-of-distribution TUT Mono-to-Binaural dataset. Our results highlight the potential of pretrained generative audio models and zero-shot learning to unlock robust binaural audio synthesis. |
| title | Zero-Shot Mono-to-Binaural Speech Synthesis |
| topic | Sound Machine Learning Audio and Speech Processing |
| url | https://arxiv.org/abs/2412.08356 |