Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2511.05516 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918191679143936 |
|---|---|
| author | Yan, Canxiang Jin, Chunxiang Huang, Dawei Yu, Haibing Peng, Han Zhan, Hui Gao, Jie Peng, Jing Chen, Jingdong Zhou, Jun Ren, Kaimeng Yang, Ming Yang, Mingxue Xu, Qiang Zhao, Qin Xiong, Ruijie Lin, Shaoxiong Wang, Xuezhi Yuan, Yi Wu, Yifei Lyu, Yongjie He, Zhengyu Qiu, Zhihao Fang, Zhiqiang Huang, Ziyuan |
| author_facet | Yan, Canxiang Jin, Chunxiang Huang, Dawei Yu, Haibing Peng, Han Zhan, Hui Gao, Jie Peng, Jing Chen, Jingdong Zhou, Jun Ren, Kaimeng Yang, Ming Yang, Mingxue Xu, Qiang Zhao, Qin Xiong, Ruijie Lin, Shaoxiong Wang, Xuezhi Yuan, Yi Wu, Yifei Lyu, Yongjie He, Zhengyu Qiu, Zhihao Fang, Zhiqiang Huang, Ziyuan |
| contents | Existing speech models suffer from competing requirements on token representations by understanding and generation tasks. This discrepancy in representation prevents speech language models from performing instruction-based free-form editing. To solve this challenge, we introduce a novel framework that unifies speech understanding, generation, and editing. The core of our unified model is a unified continuous speech tokenizer MingTok-Audio, the first continuous tokenizer to effectively integrate semantic and acoustic features, which makes it suitable for both understanding and generation tasks. Based on this unified continuous audio tokenizer, we developed the speech language model Ming-UniAudio, which achieved a balance between generation and understanding capabilities. Ming-UniAudio sets new state-of-the-art (SOTA) records on 8 out of 12 metrics on the ContextASR benchmark. Notably, for Chinese voice cloning, it achieves a highly competitive Seed-TTS-WER of 0.95. Leveraging this foundational model, we further trained a dedicated speech editing model Ming-UniAudio-Edit, the first speech language model that enables universal, free-form speech editing guided solely by natural language instructions, handling both semantic and acoustic modifications without timestamp condition. To rigorously assess the editing capability and establish a foundation for future research, we introduce Ming-Freeform-Audio-Edit, the first comprehensive benchmark tailored for instruction-based free-form speech editing, featuring diverse scenarios and evaluation dimensions spanning semantic correctness, acoustic quality, and instruction alignment. We open-sourced the continuous audio tokenizer, the unified foundational model, and the free-form instruction-based editing model to facilitate the development of unified audio understanding, generation, and manipulation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_05516 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Ming-UniAudio: Speech LLM for Joint Understanding, Generation and Editing with Unified Representation Yan, Canxiang Jin, Chunxiang Huang, Dawei Yu, Haibing Peng, Han Zhan, Hui Gao, Jie Peng, Jing Chen, Jingdong Zhou, Jun Ren, Kaimeng Yang, Ming Yang, Mingxue Xu, Qiang Zhao, Qin Xiong, Ruijie Lin, Shaoxiong Wang, Xuezhi Yuan, Yi Wu, Yifei Lyu, Yongjie He, Zhengyu Qiu, Zhihao Fang, Zhiqiang Huang, Ziyuan Computation and Language Artificial Intelligence Sound Audio and Speech Processing Existing speech models suffer from competing requirements on token representations by understanding and generation tasks. This discrepancy in representation prevents speech language models from performing instruction-based free-form editing. To solve this challenge, we introduce a novel framework that unifies speech understanding, generation, and editing. The core of our unified model is a unified continuous speech tokenizer MingTok-Audio, the first continuous tokenizer to effectively integrate semantic and acoustic features, which makes it suitable for both understanding and generation tasks. Based on this unified continuous audio tokenizer, we developed the speech language model Ming-UniAudio, which achieved a balance between generation and understanding capabilities. Ming-UniAudio sets new state-of-the-art (SOTA) records on 8 out of 12 metrics on the ContextASR benchmark. Notably, for Chinese voice cloning, it achieves a highly competitive Seed-TTS-WER of 0.95. Leveraging this foundational model, we further trained a dedicated speech editing model Ming-UniAudio-Edit, the first speech language model that enables universal, free-form speech editing guided solely by natural language instructions, handling both semantic and acoustic modifications without timestamp condition. To rigorously assess the editing capability and establish a foundation for future research, we introduce Ming-Freeform-Audio-Edit, the first comprehensive benchmark tailored for instruction-based free-form speech editing, featuring diverse scenarios and evaluation dimensions spanning semantic correctness, acoustic quality, and instruction alignment. We open-sourced the continuous audio tokenizer, the unified foundational model, and the free-form instruction-based editing model to facilitate the development of unified audio understanding, generation, and manipulation. |
| title | Ming-UniAudio: Speech LLM for Joint Understanding, Generation and Editing with Unified Representation |
| topic | Computation and Language Artificial Intelligence Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2511.05516 |