Enhance Temporal Relations in Audio Captioning with Sound Event Detection
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909260867174400 |
|---|---|
| author | Xie, Zeyu Xu, Xuenan Wu, Mengyue Yu, Kai |
| author_facet | Xie, Zeyu Xu, Xuenan Wu, Mengyue Yu, Kai |
| contents | Automated audio captioning aims at generating natural language descriptions for given audio clips, not only detecting and classifying sounds, but also summarizing the relationships between audio events. Recent research advances in audio captioning have introduced additional guidance to improve the accuracy of audio events in generated sentences. However, temporal relations between audio events have received little attention while revealing complex relations is a key component in summarizing audio content. Therefore, this paper aims to better capture temporal relationships in caption generation with sound event detection (SED), a task that locates events' timestamps. We investigate the best approach to integrate temporal information in a captioning model and propose a temporal tag system to transform the timestamps into comprehensible relations. Results evaluated by the proposed temporal metrics suggest that great improvement is achieved in terms of temporal relation generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2306_01533 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Enhance Temporal Relations in Audio Captioning with Sound Event Detection Xie, Zeyu Xu, Xuenan Wu, Mengyue Yu, Kai Sound Audio and Speech Processing Automated audio captioning aims at generating natural language descriptions for given audio clips, not only detecting and classifying sounds, but also summarizing the relationships between audio events. Recent research advances in audio captioning have introduced additional guidance to improve the accuracy of audio events in generated sentences. However, temporal relations between audio events have received little attention while revealing complex relations is a key component in summarizing audio content. Therefore, this paper aims to better capture temporal relationships in caption generation with sound event detection (SED), a task that locates events' timestamps. We investigate the best approach to integrate temporal information in a captioning model and propose a temporal tag system to transform the timestamps into comprehensible relations. Results evaluated by the proposed temporal metrics suggest that great improvement is achieved in terms of temporal relation generation. |
| title | Enhance Temporal Relations in Audio Captioning with Sound Event Detection |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2306.01533 |