Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2507.09318 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914470292357120 |
|---|---|
| author | Zhu, Han Kang, Wei Guo, Liyong Yao, Zengwei Kuang, Fangjun Zhuang, Weiji Li, Zhaoqing Han, Zhifeng Zhang, Dong Zhang, Xin Song, Xingchen Ye, Lingxuan Lin, Long Povey, Daniel |
| author_facet | Zhu, Han Kang, Wei Guo, Liyong Yao, Zengwei Kuang, Fangjun Zhuang, Weiji Li, Zhaoqing Han, Zhifeng Zhang, Dong Zhang, Xin Song, Xingchen Ye, Lingxuan Lin, Long Povey, Daniel |
| contents | Generating spoken dialogue is inherently more complex than monologue text-to-speech (TTS), as it demands both realistic turn-taking and the maintenance of distinct speaker timbres. While existing autoregressive (AR) models have made progress, they often suffer from high inference latency and stability issues. To overcome these limitations, we propose ZipVoice-Dialog, a non-autoregressive (NAR) zero-shot spoken dialogue generation model based on flow-matching. Observing that applying vanilla flow-matching to dialogue generation leads to poor speech intelligibility and turn-taking precision, we introduce two simple yet effective methods to adapt flow-matching architectures for dialogue generation: (1) a curriculum learning strategy to ensure robust speech-text alignment, and (2) speaker-turn embeddings to govern precise speaker turn-taking. Additionally, we introduce dedicated strategies to support stereo dialogue generation. Recognizing the lack of training datasets in this field, we curate and release OpenDialog, the first large-scale (6.8k hours) open-source spoken dialogue dataset derived from in-the-wild speech data. Moreover, for fair and rigorous evaluations, we established a benchmark to comprehensively evaluate dialogue generation models. Experiments demonstrate the effectiveness of the proposed methods and dataset, showing that ZipVoice-Dialog achieves superior performance in inference speed, intelligibility, speaker turn-taking accuracy, and speaker similarity. Our code, model checkpoints, and the OpenDialog dataset are publicly available at https://github.com/k2-fsa/ZipVoice. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_09318 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matching Zhu, Han Kang, Wei Guo, Liyong Yao, Zengwei Kuang, Fangjun Zhuang, Weiji Li, Zhaoqing Han, Zhifeng Zhang, Dong Zhang, Xin Song, Xingchen Ye, Lingxuan Lin, Long Povey, Daniel Audio and Speech Processing Computation and Language Generating spoken dialogue is inherently more complex than monologue text-to-speech (TTS), as it demands both realistic turn-taking and the maintenance of distinct speaker timbres. While existing autoregressive (AR) models have made progress, they often suffer from high inference latency and stability issues. To overcome these limitations, we propose ZipVoice-Dialog, a non-autoregressive (NAR) zero-shot spoken dialogue generation model based on flow-matching. Observing that applying vanilla flow-matching to dialogue generation leads to poor speech intelligibility and turn-taking precision, we introduce two simple yet effective methods to adapt flow-matching architectures for dialogue generation: (1) a curriculum learning strategy to ensure robust speech-text alignment, and (2) speaker-turn embeddings to govern precise speaker turn-taking. Additionally, we introduce dedicated strategies to support stereo dialogue generation. Recognizing the lack of training datasets in this field, we curate and release OpenDialog, the first large-scale (6.8k hours) open-source spoken dialogue dataset derived from in-the-wild speech data. Moreover, for fair and rigorous evaluations, we established a benchmark to comprehensively evaluate dialogue generation models. Experiments demonstrate the effectiveness of the proposed methods and dataset, showing that ZipVoice-Dialog achieves superior performance in inference speed, intelligibility, speaker turn-taking accuracy, and speaker similarity. Our code, model checkpoints, and the OpenDialog dataset are publicly available at https://github.com/k2-fsa/ZipVoice. |
| title | ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matching |
| topic | Audio and Speech Processing Computation and Language |
| url | https://arxiv.org/abs/2507.09318 |