M2D-CLAP: Masked Modeling Duo Meets CLAP for Learning General-purpose Audio-Language Representation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929372044197888 |
|---|---|
| author | Niizumi, Daisuke Takeuchi, Daiki Ohishi, Yasunori Harada, Noboru Yasuda, Masahiro Tsubaki, Shunsuke Imoto, Keisuke |
| author_facet | Niizumi, Daisuke Takeuchi, Daiki Ohishi, Yasunori Harada, Noboru Yasuda, Masahiro Tsubaki, Shunsuke Imoto, Keisuke |
| contents | Contrastive language-audio pre-training (CLAP) enables zero-shot (ZS) inference of audio and exhibits promising performance in several classification tasks. However, conventional audio representations are still crucial for many tasks where ZS is not applicable (e.g., regression problems). Here, we explore a new representation, a general-purpose audio-language representation, that performs well in both ZS and transfer learning. To do so, we propose a new method, M2D-CLAP, which combines self-supervised learning Masked Modeling Duo (M2D) and CLAP. M2D learns an effective representation to model audio signals, and CLAP aligns the representation with text embedding. As a result, M2D-CLAP learns a versatile representation that allows for both ZS and transfer learning. Experiments show that M2D-CLAP performs well on linear evaluation, fine-tuning, and ZS classification with a GTZAN state-of-the-art of 75.17%, thus achieving a general-purpose audio-language representation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_02032 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | M2D-CLAP: Masked Modeling Duo Meets CLAP for Learning General-purpose Audio-Language Representation Niizumi, Daisuke Takeuchi, Daiki Ohishi, Yasunori Harada, Noboru Yasuda, Masahiro Tsubaki, Shunsuke Imoto, Keisuke Audio and Speech Processing Multimedia Sound 68T07 Contrastive language-audio pre-training (CLAP) enables zero-shot (ZS) inference of audio and exhibits promising performance in several classification tasks. However, conventional audio representations are still crucial for many tasks where ZS is not applicable (e.g., regression problems). Here, we explore a new representation, a general-purpose audio-language representation, that performs well in both ZS and transfer learning. To do so, we propose a new method, M2D-CLAP, which combines self-supervised learning Masked Modeling Duo (M2D) and CLAP. M2D learns an effective representation to model audio signals, and CLAP aligns the representation with text embedding. As a result, M2D-CLAP learns a versatile representation that allows for both ZS and transfer learning. Experiments show that M2D-CLAP performs well on linear evaluation, fine-tuning, and ZS classification with a GTZAN state-of-the-art of 75.17%, thus achieving a general-purpose audio-language representation. |
| title | M2D-CLAP: Masked Modeling Duo Meets CLAP for Learning General-purpose Audio-Language Representation |
| topic | Audio and Speech Processing Multimedia Sound 68T07 |
| url | https://arxiv.org/abs/2406.02032 |