Latent CLAP Loss for Better Foley Sound Synthesis
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917801587900416 |
|---|---|
| author | Karchkhadze, Tornike Kavaki, Hassan Salami Izadi, Mohammad Rasool Irvin, Bryce Kegler, Mikolaj Hertz, Ari Zhang, Shuo Stamenovic, Marko |
| author_facet | Karchkhadze, Tornike Kavaki, Hassan Salami Izadi, Mohammad Rasool Irvin, Bryce Kegler, Mikolaj Hertz, Ari Zhang, Shuo Stamenovic, Marko |
| contents | Foley sound generation, the art of creating audio for multimedia, has recently seen notable advancements through text-conditioned latent diffusion models. These systems use multimodal text-audio representation models, such as Contrastive Language-Audio Pretraining (CLAP), whose objective is to map corresponding audio and text prompts into a joint embedding space. AudioLDM, a text-to-audio model, was the winner of the DCASE2023 task 7 Foley sound synthesis challenge. The winning system fine-tuned the model for specific audio classes and applied a post-filtering method using CLAP similarity scores between output audio and input text at inference time, requiring the generation of extra samples, thus reducing data generation efficiency. We introduce a new loss term to enhance Foley sound generation in AudioLDM without post-filtering. This loss term uses a new module based on the CLAP mode-Latent CLAP encode-to align the latent diffusion output with real audio in a shared CLAP embedding space. Our experiments demonstrate that our method effectively reduces the Frechet Audio Distance (FAD) score of the generated audio and eliminates the need for post-filtering, thus enhancing generation efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_12182 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Latent CLAP Loss for Better Foley Sound Synthesis Karchkhadze, Tornike Kavaki, Hassan Salami Izadi, Mohammad Rasool Irvin, Bryce Kegler, Mikolaj Hertz, Ari Zhang, Shuo Stamenovic, Marko Audio and Speech Processing Foley sound generation, the art of creating audio for multimedia, has recently seen notable advancements through text-conditioned latent diffusion models. These systems use multimodal text-audio representation models, such as Contrastive Language-Audio Pretraining (CLAP), whose objective is to map corresponding audio and text prompts into a joint embedding space. AudioLDM, a text-to-audio model, was the winner of the DCASE2023 task 7 Foley sound synthesis challenge. The winning system fine-tuned the model for specific audio classes and applied a post-filtering method using CLAP similarity scores between output audio and input text at inference time, requiring the generation of extra samples, thus reducing data generation efficiency. We introduce a new loss term to enhance Foley sound generation in AudioLDM without post-filtering. This loss term uses a new module based on the CLAP mode-Latent CLAP encode-to align the latent diffusion output with real audio in a shared CLAP embedding space. Our experiments demonstrate that our method effectively reduces the Frechet Audio Distance (FAD) score of the generated audio and eliminates the need for post-filtering, thus enhancing generation efficiency. |
| title | Latent CLAP Loss for Better Foley Sound Synthesis |
| topic | Audio and Speech Processing |
| url | https://arxiv.org/abs/2403.12182 |