MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910762685956096 |
|---|---|
| author | Li, Yizhi Yuan, Ruibin Zhang, Ge Ma, Yinghao Chen, Xingran Yin, Hanzhi Xiao, Chenghao Lin, Chenghua Ragni, Anton Benetos, Emmanouil Gyenge, Norbert Dannenberg, Roger Liu, Ruibo Chen, Wenhu Xia, Gus Shi, Yemin Huang, Wenhao Wang, Zili Guo, Yike Fu, Jie |
| author_facet | Li, Yizhi Yuan, Ruibin Zhang, Ge Ma, Yinghao Chen, Xingran Yin, Hanzhi Xiao, Chenghao Lin, Chenghua Ragni, Anton Benetos, Emmanouil Gyenge, Norbert Dannenberg, Roger Liu, Ruibo Chen, Wenhu Xia, Gus Shi, Yemin Huang, Wenhao Wang, Zili Guo, Yike Fu, Jie |
| contents | Self-supervised learning (SSL) has recently emerged as a promising paradigm for training generalisable models on large-scale data in the fields of vision, text, and speech. Although SSL has been proven effective in speech and audio, its application to music audio has yet to be thoroughly explored. This is partially due to the distinctive challenges associated with modelling musical knowledge, particularly tonal and pitched characteristics of music. To address this research gap, we propose an acoustic Music undERstanding model with large-scale self-supervised Training (MERT), which incorporates teacher models to provide pseudo labels in the masked language modelling (MLM) style acoustic pre-training. In our exploration, we identified an effective combination of teacher models, which outperforms conventional speech and audio approaches in terms of performance. This combination includes an acoustic teacher based on Residual Vector Quantisation - Variational AutoEncoder (RVQ-VAE) and a musical teacher based on the Constant-Q Transform (CQT). Furthermore, we explore a wide range of settings to overcome the instability in acoustic language model pre-training, which allows our designed paradigm to scale from 95M to 330M parameters. Experimental results indicate that our model can generalise and perform well on 14 music understanding tasks and attain state-of-the-art (SOTA) overall scores. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2306_00107 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training Li, Yizhi Yuan, Ruibin Zhang, Ge Ma, Yinghao Chen, Xingran Yin, Hanzhi Xiao, Chenghao Lin, Chenghua Ragni, Anton Benetos, Emmanouil Gyenge, Norbert Dannenberg, Roger Liu, Ruibo Chen, Wenhu Xia, Gus Shi, Yemin Huang, Wenhao Wang, Zili Guo, Yike Fu, Jie Sound Artificial Intelligence Computation and Language Machine Learning Audio and Speech Processing Self-supervised learning (SSL) has recently emerged as a promising paradigm for training generalisable models on large-scale data in the fields of vision, text, and speech. Although SSL has been proven effective in speech and audio, its application to music audio has yet to be thoroughly explored. This is partially due to the distinctive challenges associated with modelling musical knowledge, particularly tonal and pitched characteristics of music. To address this research gap, we propose an acoustic Music undERstanding model with large-scale self-supervised Training (MERT), which incorporates teacher models to provide pseudo labels in the masked language modelling (MLM) style acoustic pre-training. In our exploration, we identified an effective combination of teacher models, which outperforms conventional speech and audio approaches in terms of performance. This combination includes an acoustic teacher based on Residual Vector Quantisation - Variational AutoEncoder (RVQ-VAE) and a musical teacher based on the Constant-Q Transform (CQT). Furthermore, we explore a wide range of settings to overcome the instability in acoustic language model pre-training, which allows our designed paradigm to scale from 95M to 330M parameters. Experimental results indicate that our model can generalise and perform well on 14 music understanding tasks and attain state-of-the-art (SOTA) overall scores. |
| title | MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training |
| topic | Sound Artificial Intelligence Computation and Language Machine Learning Audio and Speech Processing |
| url | https://arxiv.org/abs/2306.00107 |