Improving Text-To-Audio Models with Synthetic Captions
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929414302859264 |
|---|---|
| author | Kong, Zhifeng Lee, Sang-gil Ghosal, Deepanway Majumder, Navonil Mehrish, Ambuj Valle, Rafael Poria, Soujanya Catanzaro, Bryan |
| author_facet | Kong, Zhifeng Lee, Sang-gil Ghosal, Deepanway Majumder, Navonil Mehrish, Ambuj Valle, Rafael Poria, Soujanya Catanzaro, Bryan |
| contents | It is an open challenge to obtain high quality training data, especially captions, for text-to-audio models. Although prior methods have leveraged \textit{text-only language models} to augment and improve captions, such methods have limitations related to scale and coherence between audio and captions. In this work, we propose an audio captioning pipeline that uses an \textit{audio language model} to synthesize accurate and diverse captions for audio at scale. We leverage this pipeline to produce a dataset of synthetic captions for AudioSet, named \texttt{AF-AudioSet}, and then evaluate the benefit of pre-training text-to-audio models on these synthetic captions. Through systematic evaluations on AudioCaps and MusicCaps, we find leveraging our pipeline and synthetic captions leads to significant improvements on audio generation quality, achieving a new \textit{state-of-the-art}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_15487 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Improving Text-To-Audio Models with Synthetic Captions Kong, Zhifeng Lee, Sang-gil Ghosal, Deepanway Majumder, Navonil Mehrish, Ambuj Valle, Rafael Poria, Soujanya Catanzaro, Bryan Computation and Language Machine Learning Sound Audio and Speech Processing It is an open challenge to obtain high quality training data, especially captions, for text-to-audio models. Although prior methods have leveraged \textit{text-only language models} to augment and improve captions, such methods have limitations related to scale and coherence between audio and captions. In this work, we propose an audio captioning pipeline that uses an \textit{audio language model} to synthesize accurate and diverse captions for audio at scale. We leverage this pipeline to produce a dataset of synthetic captions for AudioSet, named \texttt{AF-AudioSet}, and then evaluate the benefit of pre-training text-to-audio models on these synthetic captions. Through systematic evaluations on AudioCaps and MusicCaps, we find leveraging our pipeline and synthetic captions leads to significant improvements on audio generation quality, achieving a new \textit{state-of-the-art}. |
| title | Improving Text-To-Audio Models with Synthetic Captions |
| topic | Computation and Language Machine Learning Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2406.15487 |