Towards Diverse and Efficient Audio Captioning via Diffusion Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910978750283776 |
|---|---|
| author | Xu, Manjie Li, Chenxing Tu, Xinyi Ren, Yong Fu, Ruibo Liang, Wei Yu, Dong |
| author_facet | Xu, Manjie Li, Chenxing Tu, Xinyi Ren, Yong Fu, Ruibo Liang, Wei Yu, Dong |
| contents | We introduce Diffusion-based Audio Captioning (DAC), a non-autoregressive diffusion model tailored for diverse and efficient audio captioning. Although existing captioning models relying on language backbones have achieved remarkable success in various captioning tasks, their insufficient performance in terms of generation speed and diversity impede progress in audio understanding and multimedia applications. Our diffusion-based framework offers unique advantages stemming from its inherent stochasticity and holistic context modeling in captioning. Through rigorous evaluation, we demonstrate that DAC not only achieves SOTA performance levels compared to existing benchmarks in the caption quality, but also significantly outperforms them in terms of generation speed and diversity. The success of DAC illustrates that text generation can also be seamlessly integrated with audio and visual generation tasks using a diffusion backbone, paving the way for a unified, audio-related generative model across different modalities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_09401 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Diverse and Efficient Audio Captioning via Diffusion Models Xu, Manjie Li, Chenxing Tu, Xinyi Ren, Yong Fu, Ruibo Liang, Wei Yu, Dong Computation and Language We introduce Diffusion-based Audio Captioning (DAC), a non-autoregressive diffusion model tailored for diverse and efficient audio captioning. Although existing captioning models relying on language backbones have achieved remarkable success in various captioning tasks, their insufficient performance in terms of generation speed and diversity impede progress in audio understanding and multimedia applications. Our diffusion-based framework offers unique advantages stemming from its inherent stochasticity and holistic context modeling in captioning. Through rigorous evaluation, we demonstrate that DAC not only achieves SOTA performance levels compared to existing benchmarks in the caption quality, but also significantly outperforms them in terms of generation speed and diversity. The success of DAC illustrates that text generation can also be seamlessly integrated with audio and visual generation tasks using a diffusion backbone, paving the way for a unified, audio-related generative model across different modalities. |
| title | Towards Diverse and Efficient Audio Captioning via Diffusion Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2409.09401 |