T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretraining
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913334229467136 |
|---|---|
| author | Yuan, Yi Chen, Zhuo Liu, Xubo Liu, Haohe Xu, Xuenan Jia, Dongya Chen, Yuanzhe Plumbley, Mark D. Wang, Wenwu |
| author_facet | Yuan, Yi Chen, Zhuo Liu, Xubo Liu, Haohe Xu, Xuenan Jia, Dongya Chen, Yuanzhe Plumbley, Mark D. Wang, Wenwu |
| contents | Contrastive language-audio pretraining~(CLAP) has been developed to align the representations of audio and language, achieving remarkable performance in retrieval and classification tasks. However, current CLAP struggles to capture temporal information within audio and text features, presenting substantial limitations for tasks such as audio retrieval and generation. To address this gap, we introduce T-CLAP, a temporal-enhanced CLAP model. We use Large Language Models~(LLMs) and mixed-up strategies to generate temporal-contrastive captions for audio clips from extensive audio-text datasets. Subsequently, a new temporal-focused contrastive loss is designed to fine-tune the CLAP model by incorporating these synthetic data. We conduct comprehensive experiments and analysis in multiple downstream tasks. T-CLAP shows improved capability in capturing the temporal relationship of sound events and outperforms state-of-the-art models by a significant margin. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_17806 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretraining Yuan, Yi Chen, Zhuo Liu, Xubo Liu, Haohe Xu, Xuenan Jia, Dongya Chen, Yuanzhe Plumbley, Mark D. Wang, Wenwu Sound Computation and Language Machine Learning Audio and Speech Processing Contrastive language-audio pretraining~(CLAP) has been developed to align the representations of audio and language, achieving remarkable performance in retrieval and classification tasks. However, current CLAP struggles to capture temporal information within audio and text features, presenting substantial limitations for tasks such as audio retrieval and generation. To address this gap, we introduce T-CLAP, a temporal-enhanced CLAP model. We use Large Language Models~(LLMs) and mixed-up strategies to generate temporal-contrastive captions for audio clips from extensive audio-text datasets. Subsequently, a new temporal-focused contrastive loss is designed to fine-tune the CLAP model by incorporating these synthetic data. We conduct comprehensive experiments and analysis in multiple downstream tasks. T-CLAP shows improved capability in capturing the temporal relationship of sound events and outperforms state-of-the-art models by a significant margin. |
| title | T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretraining |
| topic | Sound Computation and Language Machine Learning Audio and Speech Processing |
| url | https://arxiv.org/abs/2404.17806 |