StreamVC: Real-Time Low-Latency Voice Conversion
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913187705651200 |
|---|---|
| author | Yang, Yang Kartynnik, Yury Li, Yunpeng Tang, Jiuqiang Li, Xing Sung, George Grundmann, Matthias |
| author_facet | Yang, Yang Kartynnik, Yury Li, Yunpeng Tang, Jiuqiang Li, Xing Sung, George Grundmann, Matthias |
| contents | We present StreamVC, a streaming voice conversion solution that preserves the content and prosody of any source speech while matching the voice timbre from any target speech. Unlike previous approaches, StreamVC produces the resulting waveform at low latency from the input signal even on a mobile platform, making it applicable to real-time communication scenarios like calls and video conferencing, and addressing use cases such as voice anonymization in these scenarios. Our design leverages the architecture and training strategy of the SoundStream neural audio codec for lightweight high-quality speech synthesis. We demonstrate the feasibility of learning soft speech units causally, as well as the effectiveness of supplying whitened fundamental frequency information to improve pitch stability without leaking the source timbre information. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_03078 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | StreamVC: Real-Time Low-Latency Voice Conversion Yang, Yang Kartynnik, Yury Li, Yunpeng Tang, Jiuqiang Li, Xing Sung, George Grundmann, Matthias Audio and Speech Processing Machine Learning Sound We present StreamVC, a streaming voice conversion solution that preserves the content and prosody of any source speech while matching the voice timbre from any target speech. Unlike previous approaches, StreamVC produces the resulting waveform at low latency from the input signal even on a mobile platform, making it applicable to real-time communication scenarios like calls and video conferencing, and addressing use cases such as voice anonymization in these scenarios. Our design leverages the architecture and training strategy of the SoundStream neural audio codec for lightweight high-quality speech synthesis. We demonstrate the feasibility of learning soft speech units causally, as well as the effectiveness of supplying whitened fundamental frequency information to improve pitch stability without leaking the source timbre information. |
| title | StreamVC: Real-Time Low-Latency Voice Conversion |
| topic | Audio and Speech Processing Machine Learning Sound |
| url | https://arxiv.org/abs/2401.03078 |