Latent Diffusion Bridges for Unsupervised Musical Audio Timbre Transfer
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912179036356608 |
|---|---|
| author | Mancusi, Michele Halychanskyi, Yurii Cheuk, Kin Wai Moliner, Eloi Lai, Chieh-Hsin Uhlich, Stefan Koo, Junghyun Martínez-Ramírez, Marco A. Liao, Wei-Hsiang Fabbro, Giorgio Mitsufuji, Yuki |
| author_facet | Mancusi, Michele Halychanskyi, Yurii Cheuk, Kin Wai Moliner, Eloi Lai, Chieh-Hsin Uhlich, Stefan Koo, Junghyun Martínez-Ramírez, Marco A. Liao, Wei-Hsiang Fabbro, Giorgio Mitsufuji, Yuki |
| contents | Music timbre transfer is a challenging task that involves modifying the timbral characteristics of an audio signal while preserving its melodic structure. In this paper, we propose a novel method based on dual diffusion bridges, trained using the CocoChorales Dataset, which consists of unpaired monophonic single-instrument audio data. Each diffusion model is trained on a specific instrument with a Gaussian prior. During inference, a model is designated as the source model to map the input audio to its corresponding Gaussian prior, and another model is designated as the target model to reconstruct the target audio from this Gaussian prior, thereby facilitating timbre transfer. We compare our approach against existing unsupervised timbre transfer models such as VAEGAN and Gaussian Flow Bridges (GFB). Experimental results demonstrate that our method achieves both better Fréchet Audio Distance (FAD) and melody preservation, as reflected by lower pitch distances (DPD) compared to VAEGAN and GFB. Additionally, we discover that the noise level from the Gaussian prior, $σ$, can be adjusted to control the degree of melody preservation and amount of timbre transferred. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_06096 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Latent Diffusion Bridges for Unsupervised Musical Audio Timbre Transfer Mancusi, Michele Halychanskyi, Yurii Cheuk, Kin Wai Moliner, Eloi Lai, Chieh-Hsin Uhlich, Stefan Koo, Junghyun Martínez-Ramírez, Marco A. Liao, Wei-Hsiang Fabbro, Giorgio Mitsufuji, Yuki Sound Artificial Intelligence Information Retrieval Audio and Speech Processing Music timbre transfer is a challenging task that involves modifying the timbral characteristics of an audio signal while preserving its melodic structure. In this paper, we propose a novel method based on dual diffusion bridges, trained using the CocoChorales Dataset, which consists of unpaired monophonic single-instrument audio data. Each diffusion model is trained on a specific instrument with a Gaussian prior. During inference, a model is designated as the source model to map the input audio to its corresponding Gaussian prior, and another model is designated as the target model to reconstruct the target audio from this Gaussian prior, thereby facilitating timbre transfer. We compare our approach against existing unsupervised timbre transfer models such as VAEGAN and Gaussian Flow Bridges (GFB). Experimental results demonstrate that our method achieves both better Fréchet Audio Distance (FAD) and melody preservation, as reflected by lower pitch distances (DPD) compared to VAEGAN and GFB. Additionally, we discover that the noise level from the Gaussian prior, $σ$, can be adjusted to control the degree of melody preservation and amount of timbre transferred. |
| title | Latent Diffusion Bridges for Unsupervised Musical Audio Timbre Transfer |
| topic | Sound Artificial Intelligence Information Retrieval Audio and Speech Processing |
| url | https://arxiv.org/abs/2409.06096 |