Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.10708 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914509047726080 |
|---|---|
| author | Tian, Zeyue Yang, Binxin Liu, Zhaoyang Zhang, Jiexuan Yuan, Ruibin Yin, Hubery Chen, Qifeng Li, Chen Lyu, Jing Xue, Wei Guo, Yike |
| author_facet | Tian, Zeyue Yang, Binxin Liu, Zhaoyang Zhang, Jiexuan Yuan, Ruibin Yin, Hubery Chen, Qifeng Li, Chen Lyu, Jing Xue, Wei Guo, Yike |
| contents | Recent progress in multimodal models has spurred rapid advances in audio understanding, generation, and editing. However, these capabilities are typically addressed by specialized models, leaving the development of a truly unified framework that can seamlessly integrate all three tasks underexplored. While some pioneering works have explored unifying audio understanding and generation, they often remain confined to specific domains. To address this, we introduce Audio-Omni, the first end-to-end framework to unify generation and editing across general sound, music, and speech domains, with integrated multi-modal understanding capabilities. Our architecture synergizes a frozen Multimodal Large Language Model for high-level reasoning with a trainable Diffusion Transformer for high-fidelity synthesis. To overcome the critical data scarcity in audio editing, we construct AudioEdit, a new large-scale dataset comprising over one million meticulously curated editing pairs. Extensive experiments demonstrate that Audio-Omni achieves state-of-the-art performance across a suite of benchmarks, outperforming prior unified approaches while achieving performance on par with or superior to specialized expert models. Beyond its core capabilities, Audio-Omni exhibits remarkable inherited capabilities, including knowledge-augmented reasoning generation, in-context generation, and zero-shot cross-lingual control for audio generation, highlighting a promising direction toward universal generative audio intelligence. The code, model, and dataset will be publicly released on https://zeyuet.github.io/Audio-Omni. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_10708 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing Tian, Zeyue Yang, Binxin Liu, Zhaoyang Zhang, Jiexuan Yuan, Ruibin Yin, Hubery Chen, Qifeng Li, Chen Lyu, Jing Xue, Wei Guo, Yike Sound Artificial Intelligence Computer Vision and Pattern Recognition Multimedia Recent progress in multimodal models has spurred rapid advances in audio understanding, generation, and editing. However, these capabilities are typically addressed by specialized models, leaving the development of a truly unified framework that can seamlessly integrate all three tasks underexplored. While some pioneering works have explored unifying audio understanding and generation, they often remain confined to specific domains. To address this, we introduce Audio-Omni, the first end-to-end framework to unify generation and editing across general sound, music, and speech domains, with integrated multi-modal understanding capabilities. Our architecture synergizes a frozen Multimodal Large Language Model for high-level reasoning with a trainable Diffusion Transformer for high-fidelity synthesis. To overcome the critical data scarcity in audio editing, we construct AudioEdit, a new large-scale dataset comprising over one million meticulously curated editing pairs. Extensive experiments demonstrate that Audio-Omni achieves state-of-the-art performance across a suite of benchmarks, outperforming prior unified approaches while achieving performance on par with or superior to specialized expert models. Beyond its core capabilities, Audio-Omni exhibits remarkable inherited capabilities, including knowledge-augmented reasoning generation, in-context generation, and zero-shot cross-lingual control for audio generation, highlighting a promising direction toward universal generative audio intelligence. The code, model, and dataset will be publicly released on https://zeyuet.github.io/Audio-Omni. |
| title | Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing |
| topic | Sound Artificial Intelligence Computer Vision and Pattern Recognition Multimedia |
| url | https://arxiv.org/abs/2604.10708 |