Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.31530 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910272544833536 |
|---|---|
| author | Li, Zhaoqing Xu, Haoning Su, Jingran Liu, Yaofang Rao, Zhefan Wang, Huimeng Deng, Jiajun Wang, Tianzi Jin, Zengrui Liu, Rui Che, Haoxuan Liu, Xunying |
| author_facet | Li, Zhaoqing Xu, Haoning Su, Jingran Liu, Yaofang Rao, Zhefan Wang, Huimeng Deng, Jiajun Wang, Tianzi Jin, Zengrui Liu, Rui Che, Haoxuan Liu, Xunying |
| contents | We present UNISON, a latent diffusion framework that unifies speech generation, sound generation, and audio editing within a single model. A single model handles text-to-audio, text-to-speech, zero-shot speaker cloning, mixed speech-and-sound generation, scene-level audio editing, speech-in-scene editing, and timed temporal composition, all of which share a single set of weights. Our architecture features two core designs: (1) Layer-wise deep LLM fusion, which injects hidden states from uniformly sampled layers of a frozen MLLM into corresponding MM-DiT blocks via learned projections, providing depth-matched semantic conditioning that improves instruction following over single-layer baselines; and (2) a unified multi-task architecture where task identity is encoded solely by a channel-wise mask and source audio is provided through VAE-encoded channel concatenation. Training is stabilized by an online GPU-side multi-task data synthesis pipeline with task-homogeneous batching and a two-stage curriculum. With 621M--732M trainable parameters, UNISON achieves results competitive with or exceeding task-specialist models across evaluated domains, while being roughly $4\times$ smaller than comparable unified systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_31530 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | UNISON: A Unified Sound Generation and Editing Framework via Deep LLM Fusion Li, Zhaoqing Xu, Haoning Su, Jingran Liu, Yaofang Rao, Zhefan Wang, Huimeng Deng, Jiajun Wang, Tianzi Jin, Zengrui Liu, Rui Che, Haoxuan Liu, Xunying Audio and Speech Processing Sound We present UNISON, a latent diffusion framework that unifies speech generation, sound generation, and audio editing within a single model. A single model handles text-to-audio, text-to-speech, zero-shot speaker cloning, mixed speech-and-sound generation, scene-level audio editing, speech-in-scene editing, and timed temporal composition, all of which share a single set of weights. Our architecture features two core designs: (1) Layer-wise deep LLM fusion, which injects hidden states from uniformly sampled layers of a frozen MLLM into corresponding MM-DiT blocks via learned projections, providing depth-matched semantic conditioning that improves instruction following over single-layer baselines; and (2) a unified multi-task architecture where task identity is encoded solely by a channel-wise mask and source audio is provided through VAE-encoded channel concatenation. Training is stabilized by an online GPU-side multi-task data synthesis pipeline with task-homogeneous batching and a two-stage curriculum. With 621M--732M trainable parameters, UNISON achieves results competitive with or exceeding task-specialist models across evaluated domains, while being roughly $4\times$ smaller than comparable unified systems. |
| title | UNISON: A Unified Sound Generation and Editing Framework via Deep LLM Fusion |
| topic | Audio and Speech Processing Sound |
| url | https://arxiv.org/abs/2605.31530 |