Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.14340 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909303025172480 |
|---|---|
| author | Ma, Yinghao Øland, Anders Ragni, Anton Del Sette, Bleiz MacSen Saitis, Charalampos Donahue, Chris Lin, Chenghua Plachouras, Christos Benetos, Emmanouil Shatri, Elona Morreale, Fabio Zhang, Ge Fazekas, György Xia, Gus Zhang, Huan Manco, Ilaria Huang, Jiawen Guinot, Julien Lin, Liwei Marinelli, Luca Lam, Max W. Y. Sharma, Megha Kong, Qiuqiang Dannenberg, Roger B. Yuan, Ruibin Wu, Shangda Wu, Shih-Lun Dai, Shuqi Lei, Shun Kang, Shiyin Dixon, Simon Chen, Wenhu Huang, Wenhao Du, Xingjian Qu, Xingwei Tan, Xu Li, Yizhi Tian, Zeyue Wu, Zhiyong Wu, Zhizheng Ma, Ziyang Wang, Ziyu |
| author_facet | Ma, Yinghao Øland, Anders Ragni, Anton Del Sette, Bleiz MacSen Saitis, Charalampos Donahue, Chris Lin, Chenghua Plachouras, Christos Benetos, Emmanouil Shatri, Elona Morreale, Fabio Zhang, Ge Fazekas, György Xia, Gus Zhang, Huan Manco, Ilaria Huang, Jiawen Guinot, Julien Lin, Liwei Marinelli, Luca Lam, Max W. Y. Sharma, Megha Kong, Qiuqiang Dannenberg, Roger B. Yuan, Ruibin Wu, Shangda Wu, Shih-Lun Dai, Shuqi Lei, Shun Kang, Shiyin Dixon, Simon Chen, Wenhu Huang, Wenhao Du, Xingjian Qu, Xingwei Tan, Xu Li, Yizhi Tian, Zeyue Wu, Zhiyong Wu, Zhizheng Ma, Ziyang Wang, Ziyu |
| contents | In recent years, foundation models (FMs) such as large language models (LLMs) and latent diffusion models (LDMs) have profoundly impacted diverse sectors, including music. This comprehensive review examines state-of-the-art (SOTA) pre-trained models and foundation models in music, spanning from representation learning, generative learning and multimodal learning. We first contextualise the significance of music in various industries and trace the evolution of AI in music. By delineating the modalities targeted by foundation models, we discover many of the music representations are underexplored in FM development. Then, emphasis is placed on the lack of versatility of previous methods on diverse music applications, along with the potential of FMs in music understanding, generation and medical application. By comprehensively exploring the details of the model pre-training paradigm, architectural choices, tokenisation, finetuning methodologies and controllability, we emphasise the important topics that should have been well explored, like instruction tuning and in-context learning, scaling law and emergent ability, as well as long-sequence modelling etc. A dedicated section presents insights into music agents, accompanied by a thorough analysis of datasets and evaluations essential for pre-training and downstream tasks. Finally, by underscoring the vital importance of ethical considerations, we advocate that following research on FM for music should focus more on such issues as interpretability, transparency, human responsibility, and copyright issues. The paper offers insights into future challenges and trends on FMs for music, aiming to shape the trajectory of human-AI collaboration in the music realm. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_14340 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Foundation Models for Music: A Survey Ma, Yinghao Øland, Anders Ragni, Anton Del Sette, Bleiz MacSen Saitis, Charalampos Donahue, Chris Lin, Chenghua Plachouras, Christos Benetos, Emmanouil Shatri, Elona Morreale, Fabio Zhang, Ge Fazekas, György Xia, Gus Zhang, Huan Manco, Ilaria Huang, Jiawen Guinot, Julien Lin, Liwei Marinelli, Luca Lam, Max W. Y. Sharma, Megha Kong, Qiuqiang Dannenberg, Roger B. Yuan, Ruibin Wu, Shangda Wu, Shih-Lun Dai, Shuqi Lei, Shun Kang, Shiyin Dixon, Simon Chen, Wenhu Huang, Wenhao Du, Xingjian Qu, Xingwei Tan, Xu Li, Yizhi Tian, Zeyue Wu, Zhiyong Wu, Zhizheng Ma, Ziyang Wang, Ziyu Sound Artificial Intelligence Computation and Language Machine Learning Audio and Speech Processing In recent years, foundation models (FMs) such as large language models (LLMs) and latent diffusion models (LDMs) have profoundly impacted diverse sectors, including music. This comprehensive review examines state-of-the-art (SOTA) pre-trained models and foundation models in music, spanning from representation learning, generative learning and multimodal learning. We first contextualise the significance of music in various industries and trace the evolution of AI in music. By delineating the modalities targeted by foundation models, we discover many of the music representations are underexplored in FM development. Then, emphasis is placed on the lack of versatility of previous methods on diverse music applications, along with the potential of FMs in music understanding, generation and medical application. By comprehensively exploring the details of the model pre-training paradigm, architectural choices, tokenisation, finetuning methodologies and controllability, we emphasise the important topics that should have been well explored, like instruction tuning and in-context learning, scaling law and emergent ability, as well as long-sequence modelling etc. A dedicated section presents insights into music agents, accompanied by a thorough analysis of datasets and evaluations essential for pre-training and downstream tasks. Finally, by underscoring the vital importance of ethical considerations, we advocate that following research on FM for music should focus more on such issues as interpretability, transparency, human responsibility, and copyright issues. The paper offers insights into future challenges and trends on FMs for music, aiming to shape the trajectory of human-AI collaboration in the music realm. |
| title | Foundation Models for Music: A Survey |
| topic | Sound Artificial Intelligence Computation and Language Machine Learning Audio and Speech Processing |
| url | https://arxiv.org/abs/2408.14340 |