UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908990846271488 |
|---|---|
| author | Qiang, Chunyu Wang, Xiaopeng Yin, Kang Liang, Yuzhe Guo, Yuxin Ma, Teng Zhang, Ziyu Wang, Tianrui Gong, Cheng Chen, Yushen Fu, Ruibo Zhang, Chen Wang, Longbiao Dang, Jianwu |
| author_facet | Qiang, Chunyu Wang, Xiaopeng Yin, Kang Liang, Yuzhe Guo, Yuxin Ma, Teng Zhang, Ziyu Wang, Tianrui Gong, Cheng Chen, Yushen Fu, Ruibo Zhang, Chen Wang, Longbiao Dang, Jianwu |
| contents | Generative audio modeling has largely been fragmented into specialized tasks, text-to-speech (TTS), text-to-music (TTM), and text-to-audio (TTA), each operating under heterogeneous control paradigms. Unifying these modalities remains a fundamental challenge due to the intrinsic dissonance between structured semantic representations (speech/music) and unstructured acoustic textures (sound effects). In this paper, we introduce UniSonate, a unified flow-matching framework capable of synthesizing speech, music, and sound effects through a standardized, reference-free natural language instruction interface. To reconcile structural disparities, we propose a novel dynamic token injection mechanism that projects unstructured environmental sounds into a structured temporal latent space, enabling precise duration control within a phoneme-driven Multimodal Diffusion Transformer (MM-DiT). Coupled with a multi-stage curriculum learning strategy, this approach effectively mitigates cross-modal optimization conflicts. Extensive experiments demonstrate that UniSonate achieves state-of-the-art performance in instruction-based TTS (WER 1.47%) and TTM (SongEval Coherence 3.18), while maintaining competitive fidelity in TTA. Crucially, we observe positive transfer, where joint training on diverse audio data significantly enhances structural coherence and prosodic expressiveness compared to single-task baselines. Audio samples are available at https://qiangchunyu.github.io/UniSonate/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_22209 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions Qiang, Chunyu Wang, Xiaopeng Yin, Kang Liang, Yuzhe Guo, Yuxin Ma, Teng Zhang, Ziyu Wang, Tianrui Gong, Cheng Chen, Yushen Fu, Ruibo Zhang, Chen Wang, Longbiao Dang, Jianwu Audio and Speech Processing Artificial Intelligence Computation and Language Sound Generative audio modeling has largely been fragmented into specialized tasks, text-to-speech (TTS), text-to-music (TTM), and text-to-audio (TTA), each operating under heterogeneous control paradigms. Unifying these modalities remains a fundamental challenge due to the intrinsic dissonance between structured semantic representations (speech/music) and unstructured acoustic textures (sound effects). In this paper, we introduce UniSonate, a unified flow-matching framework capable of synthesizing speech, music, and sound effects through a standardized, reference-free natural language instruction interface. To reconcile structural disparities, we propose a novel dynamic token injection mechanism that projects unstructured environmental sounds into a structured temporal latent space, enabling precise duration control within a phoneme-driven Multimodal Diffusion Transformer (MM-DiT). Coupled with a multi-stage curriculum learning strategy, this approach effectively mitigates cross-modal optimization conflicts. Extensive experiments demonstrate that UniSonate achieves state-of-the-art performance in instruction-based TTS (WER 1.47%) and TTM (SongEval Coherence 3.18), while maintaining competitive fidelity in TTA. Crucially, we observe positive transfer, where joint training on diverse audio data significantly enhances structural coherence and prosodic expressiveness compared to single-task baselines. Audio samples are available at https://qiangchunyu.github.io/UniSonate/. |
| title | UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions |
| topic | Audio and Speech Processing Artificial Intelligence Computation and Language Sound |
| url | https://arxiv.org/abs/2604.22209 |