Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.10547 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912848184082432 |
|---|---|
| author | Yang, Dongchao Xie, Yuxin Yin, Yuguo Wang, Zheyu Yi, Xiaoyu Zhu, Gongxi Weng, Xiaolong Xiong, Zihan Ma, Yingzhe Cong, Dading Liu, Jingliang Huang, Zihang Ru, Jinghan Huang, Rongjie Wan, Haoran Wang, Peixu Yu, Kuoxi Wang, Helin Liang, Liming Zhuang, Xianwei Wang, Yuanyuan Dingdong Wang Guo, Haohan Cao, Junjie Ju, Zeqian Liu, Songxiang Cao, Yuewen Weng, Heming Zou, Yuexian |
| author_facet | Yang, Dongchao Xie, Yuxin Yin, Yuguo Wang, Zheyu Yi, Xiaoyu Zhu, Gongxi Weng, Xiaolong Xiong, Zihan Ma, Yingzhe Cong, Dading Liu, Jingliang Huang, Zihang Ru, Jinghan Huang, Rongjie Wan, Haoran Wang, Peixu Yu, Kuoxi Wang, Helin Liang, Liming Zhuang, Xianwei Wang, Yuanyuan Dingdong Wang Guo, Haohan Cao, Junjie Ju, Zeqian Liu, Songxiang Cao, Yuewen Weng, Heming Zou, Yuexian |
| contents | We present a family of open-source Music Foundation Models designed to advance large-scale music understanding and generation across diverse tasks and modalities. Our framework consists of four major components: (1) HeartCLAP, an audio-text alignment model; (2) HeartTranscriptor, a robust lyric recognition model optimized for real-world music scenarios; and (3) HeartCodec, a low-frame-rate (12.5 Hz) yet high-fidelity music codec tokenizer that captures long-range musical structure while preserving fine-grained acoustic details and enabling efficient autoregressive modeling; (4) HeartMuLa, an LLM-based song generation model capable of synthesizing high-fidelity music under rich, user-controllable conditions (e.g., textual style descriptions, lyrics, and reference audio). In addition, it provides two specialized modes: (i) fine-grained musical attribute control, which allows users to specify the style of different song sections (e.g., intro, verse, chorus) using natural language prompts; and (ii) short, engaging music generation, which is suitable as background music for short videos. Lastly, HeartMuLa improves significantly when scaled to 7B parameters. For the first time, we show that a Suno-level, commercial-grade system can be reproduced using academic-scale data and GPU resources. We expect these foundation models to serve as strong baselines for future research and to facilitate practical applications in multimodal content production. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_10547 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | HeartMuLa: A Family of Open Sourced Music Foundation Models Yang, Dongchao Xie, Yuxin Yin, Yuguo Wang, Zheyu Yi, Xiaoyu Zhu, Gongxi Weng, Xiaolong Xiong, Zihan Ma, Yingzhe Cong, Dading Liu, Jingliang Huang, Zihang Ru, Jinghan Huang, Rongjie Wan, Haoran Wang, Peixu Yu, Kuoxi Wang, Helin Liang, Liming Zhuang, Xianwei Wang, Yuanyuan Dingdong Wang Guo, Haohan Cao, Junjie Ju, Zeqian Liu, Songxiang Cao, Yuewen Weng, Heming Zou, Yuexian Sound We present a family of open-source Music Foundation Models designed to advance large-scale music understanding and generation across diverse tasks and modalities. Our framework consists of four major components: (1) HeartCLAP, an audio-text alignment model; (2) HeartTranscriptor, a robust lyric recognition model optimized for real-world music scenarios; and (3) HeartCodec, a low-frame-rate (12.5 Hz) yet high-fidelity music codec tokenizer that captures long-range musical structure while preserving fine-grained acoustic details and enabling efficient autoregressive modeling; (4) HeartMuLa, an LLM-based song generation model capable of synthesizing high-fidelity music under rich, user-controllable conditions (e.g., textual style descriptions, lyrics, and reference audio). In addition, it provides two specialized modes: (i) fine-grained musical attribute control, which allows users to specify the style of different song sections (e.g., intro, verse, chorus) using natural language prompts; and (ii) short, engaging music generation, which is suitable as background music for short videos. Lastly, HeartMuLa improves significantly when scaled to 7B parameters. For the first time, we show that a Suno-level, commercial-grade system can be reproduced using academic-scale data and GPU resources. We expect these foundation models to serve as strong baselines for future research and to facilitate practical applications in multimodal content production. |
| title | HeartMuLa: A Family of Open Sourced Music Foundation Models |
| topic | Sound |
| url | https://arxiv.org/abs/2601.10547 |