CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916524174868480 |
|---|---|
| author | Zhang, Leying Qian, Yao Zhou, Long Liu, Shujie Wang, Dongmei Wang, Xiaofei Yousefi, Midia Qian, Yanmin Li, Jinyu He, Lei Zhao, Sheng Zeng, Michael |
| author_facet | Zhang, Leying Qian, Yao Zhou, Long Liu, Shujie Wang, Dongmei Wang, Xiaofei Yousefi, Midia Qian, Yanmin Li, Jinyu He, Lei Zhao, Sheng Zeng, Michael |
| contents | Recent advancements in zero-shot text-to-speech (TTS) modeling have led to significant strides in generating high-fidelity and diverse speech. However, dialogue generation, along with achieving human-like naturalness in speech, continues to be a challenge. In this paper, we introduce CoVoMix: Conversational Voice Mixture Generation, a novel model for zero-shot, human-like, multi-speaker, multi-round dialogue speech generation. CoVoMix first converts dialogue text into multiple streams of discrete tokens, with each token stream representing semantic information for individual talkers. These token streams are then fed into a flow-matching based acoustic model to generate mixed mel-spectrograms. Finally, the speech waveforms are produced using a HiFi-GAN model. Furthermore, we devise a comprehensive set of metrics for measuring the effectiveness of dialogue modeling and generation. Our experimental results show that CoVoMix can generate dialogues that are not only human-like in their naturalness and coherence but also involve multiple talkers engaging in multiple rounds of conversation. This is exemplified by instances generated in a single channel where one speaker's utterance is seamlessly mixed with another's interjections or laughter, indicating the latter's role as an attentive listener. Audio samples are available at https://aka.ms/covomix. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_06690 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations Zhang, Leying Qian, Yao Zhou, Long Liu, Shujie Wang, Dongmei Wang, Xiaofei Yousefi, Midia Qian, Yanmin Li, Jinyu He, Lei Zhao, Sheng Zeng, Michael Audio and Speech Processing Artificial Intelligence Computation and Language Machine Learning Sound Recent advancements in zero-shot text-to-speech (TTS) modeling have led to significant strides in generating high-fidelity and diverse speech. However, dialogue generation, along with achieving human-like naturalness in speech, continues to be a challenge. In this paper, we introduce CoVoMix: Conversational Voice Mixture Generation, a novel model for zero-shot, human-like, multi-speaker, multi-round dialogue speech generation. CoVoMix first converts dialogue text into multiple streams of discrete tokens, with each token stream representing semantic information for individual talkers. These token streams are then fed into a flow-matching based acoustic model to generate mixed mel-spectrograms. Finally, the speech waveforms are produced using a HiFi-GAN model. Furthermore, we devise a comprehensive set of metrics for measuring the effectiveness of dialogue modeling and generation. Our experimental results show that CoVoMix can generate dialogues that are not only human-like in their naturalness and coherence but also involve multiple talkers engaging in multiple rounds of conversation. This is exemplified by instances generated in a single channel where one speaker's utterance is seamlessly mixed with another's interjections or laughter, indicating the latter's role as an attentive listener. Audio samples are available at https://aka.ms/covomix. |
| title | CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations |
| topic | Audio and Speech Processing Artificial Intelligence Computation and Language Machine Learning Sound |
| url | https://arxiv.org/abs/2404.06690 |