A Detailed Audio-Text Data Simulation Pipeline using Single-Event Sounds
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929267741294592 |
|---|---|
| author | Xu, Xuenan Xu, Xiaohang Xie, Zeyu Zhang, Pingyue Wu, Mengyue Yu, Kai |
| author_facet | Xu, Xuenan Xu, Xiaohang Xie, Zeyu Zhang, Pingyue Wu, Mengyue Yu, Kai |
| contents | Recently, there has been an increasing focus on audio-text cross-modal learning. However, most of the existing audio-text datasets contain only simple descriptions of sound events. Compared with classification labels, the advantages of such descriptions are significantly limited. In this paper, we first analyze the detailed information that human descriptions of audio may contain beyond sound event labels. Based on the analysis, we propose an automatic pipeline for curating audio-text pairs with rich details. Leveraging the property that sounds can be mixed and concatenated in the time domain, we control details in four aspects: temporal relationship, loudness, speaker identity, and occurrence number, in simulating audio mixtures. Corresponding details are transformed into captions by large language models. Audio-text pairs with rich details in text descriptions are thereby obtained. We validate the effectiveness of our pipeline with a small amount of simulated data, demonstrating that the simulated data enables models to learn detailed audio captioning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_04594 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Detailed Audio-Text Data Simulation Pipeline using Single-Event Sounds Xu, Xuenan Xu, Xiaohang Xie, Zeyu Zhang, Pingyue Wu, Mengyue Yu, Kai Sound Audio and Speech Processing Recently, there has been an increasing focus on audio-text cross-modal learning. However, most of the existing audio-text datasets contain only simple descriptions of sound events. Compared with classification labels, the advantages of such descriptions are significantly limited. In this paper, we first analyze the detailed information that human descriptions of audio may contain beyond sound event labels. Based on the analysis, we propose an automatic pipeline for curating audio-text pairs with rich details. Leveraging the property that sounds can be mixed and concatenated in the time domain, we control details in four aspects: temporal relationship, loudness, speaker identity, and occurrence number, in simulating audio mixtures. Corresponding details are transformed into captions by large language models. Audio-text pairs with rich details in text descriptions are thereby obtained. We validate the effectiveness of our pipeline with a small amount of simulated data, demonstrating that the simulated data enables models to learn detailed audio captioning. |
| title | A Detailed Audio-Text Data Simulation Pipeline using Single-Event Sounds |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2403.04594 |