The NPU-HWC System for the ISCSLP 2024 Inspirational and Convincing Audio Generation Challenge
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910679287463936 |
|---|---|
| author | Guo, Dake Yao, Jixun Zhu, Xinfa Xia, Kangxiang Guo, Zhao Zhang, Ziyu Wang, Yao Liu, Jie Xie, Lei |
| author_facet | Guo, Dake Yao, Jixun Zhu, Xinfa Xia, Kangxiang Guo, Zhao Zhang, Ziyu Wang, Yao Liu, Jie Xie, Lei |
| contents | This paper presents the NPU-HWC system submitted to the ISCSLP 2024 Inspirational and Convincing Audio Generation Challenge 2024 (ICAGC). Our system consists of two modules: a speech generator for Track 1 and a background audio generator for Track 2. In Track 1, we employ Single-Codec to tokenize the speech into discrete tokens and use a language-model-based approach to achieve zero-shot speaking style cloning. The Single-Codec effectively decouples timbre and speaking style at the token level, reducing the acoustic modeling burden on the autoregressive language model. Additionally, we use DSPGAN to upsample 16 kHz mel-spectrograms to high-fidelity 48 kHz waveforms. In Track 2, we propose a background audio generator based on large language models (LLMs). This system produces scene-appropriate accompaniment descriptions, synthesizes background audio with Tango 2, and integrates it with the speech generated by our Track 1 system. Our submission achieves the second place and the first place in Track 1 and Track 2 respectively. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_23815 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The NPU-HWC System for the ISCSLP 2024 Inspirational and Convincing Audio Generation Challenge Guo, Dake Yao, Jixun Zhu, Xinfa Xia, Kangxiang Guo, Zhao Zhang, Ziyu Wang, Yao Liu, Jie Xie, Lei Sound Artificial Intelligence Audio and Speech Processing This paper presents the NPU-HWC system submitted to the ISCSLP 2024 Inspirational and Convincing Audio Generation Challenge 2024 (ICAGC). Our system consists of two modules: a speech generator for Track 1 and a background audio generator for Track 2. In Track 1, we employ Single-Codec to tokenize the speech into discrete tokens and use a language-model-based approach to achieve zero-shot speaking style cloning. The Single-Codec effectively decouples timbre and speaking style at the token level, reducing the acoustic modeling burden on the autoregressive language model. Additionally, we use DSPGAN to upsample 16 kHz mel-spectrograms to high-fidelity 48 kHz waveforms. In Track 2, we propose a background audio generator based on large language models (LLMs). This system produces scene-appropriate accompaniment descriptions, synthesizes background audio with Tango 2, and integrates it with the speech generated by our Track 1 system. Our submission achieves the second place and the first place in Track 1 and Track 2 respectively. |
| title | The NPU-HWC System for the ISCSLP 2024 Inspirational and Convincing Audio Generation Challenge |
| topic | Sound Artificial Intelligence Audio and Speech Processing |
| url | https://arxiv.org/abs/2410.23815 |