VQ-CTAP: Cross-Modal Fine-Grained Sequence Representation Learning for Speech Processing
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916762353664000 |
|---|---|
| author | Qiang, Chunyu Geng, Wang Zhao, Yi Fu, Ruibo Wang, Tao Gong, Cheng Wang, Tianrui Liu, Qiuyu Yi, Jiangyan Wen, Zhengqi Zhang, Chen Che, Hao Wang, Longbiao Dang, Jianwu Tao, Jianhua |
| author_facet | Qiang, Chunyu Geng, Wang Zhao, Yi Fu, Ruibo Wang, Tao Gong, Cheng Wang, Tianrui Liu, Qiuyu Yi, Jiangyan Wen, Zhengqi Zhang, Chen Che, Hao Wang, Longbiao Dang, Jianwu Tao, Jianhua |
| contents | Deep learning has brought significant improvements to the field of cross-modal representation learning. For tasks such as text-to-speech (TTS), voice conversion (VC), and automatic speech recognition (ASR), a cross-modal fine-grained (frame-level) sequence representation is desired, emphasizing the semantic content of the text modality while de-emphasizing the paralinguistic information of the speech modality. We propose a method called "Vector Quantized Contrastive Token-Acoustic Pre-training (VQ-CTAP)", which uses the cross-modal aligned sequence transcoder to bring text and speech into a joint multimodal space, learning how to connect text and speech at the frame level. The proposed VQ-CTAP is a paradigm for cross-modal sequence representation learning, offering a promising solution for fine-grained generation and recognition tasks in speech processing. The VQ-CTAP can be directly applied to VC and ASR tasks without fine-tuning or additional structures. We propose a sequence-aware semantic connector, which connects multiple frozen pre-trained modules for the TTS task, exhibiting a plug-and-play capability. We design a stepping optimization strategy to ensure effective model convergence by gradually injecting and adjusting the influence of various loss components. Furthermore, we propose a semantic-transfer-wise paralinguistic consistency loss to enhance representational capabilities, allowing the model to better generalize to unseen data and capture the nuances of paralinguistic information. In addition, VQ-CTAP achieves high-compression speech coding at a rate of 25Hz from 24kHz input waveforms, which is a 960-fold reduction in the sampling rate. The audio demo is available at https://qiangchunyu.github.io/VQCTAP/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_05758 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | VQ-CTAP: Cross-Modal Fine-Grained Sequence Representation Learning for Speech Processing Qiang, Chunyu Geng, Wang Zhao, Yi Fu, Ruibo Wang, Tao Gong, Cheng Wang, Tianrui Liu, Qiuyu Yi, Jiangyan Wen, Zhengqi Zhang, Chen Che, Hao Wang, Longbiao Dang, Jianwu Tao, Jianhua Audio and Speech Processing Artificial Intelligence Computation and Language Sound Deep learning has brought significant improvements to the field of cross-modal representation learning. For tasks such as text-to-speech (TTS), voice conversion (VC), and automatic speech recognition (ASR), a cross-modal fine-grained (frame-level) sequence representation is desired, emphasizing the semantic content of the text modality while de-emphasizing the paralinguistic information of the speech modality. We propose a method called "Vector Quantized Contrastive Token-Acoustic Pre-training (VQ-CTAP)", which uses the cross-modal aligned sequence transcoder to bring text and speech into a joint multimodal space, learning how to connect text and speech at the frame level. The proposed VQ-CTAP is a paradigm for cross-modal sequence representation learning, offering a promising solution for fine-grained generation and recognition tasks in speech processing. The VQ-CTAP can be directly applied to VC and ASR tasks without fine-tuning or additional structures. We propose a sequence-aware semantic connector, which connects multiple frozen pre-trained modules for the TTS task, exhibiting a plug-and-play capability. We design a stepping optimization strategy to ensure effective model convergence by gradually injecting and adjusting the influence of various loss components. Furthermore, we propose a semantic-transfer-wise paralinguistic consistency loss to enhance representational capabilities, allowing the model to better generalize to unseen data and capture the nuances of paralinguistic information. In addition, VQ-CTAP achieves high-compression speech coding at a rate of 25Hz from 24kHz input waveforms, which is a 960-fold reduction in the sampling rate. The audio demo is available at https://qiangchunyu.github.io/VQCTAP/ |
| title | VQ-CTAP: Cross-Modal Fine-Grained Sequence Representation Learning for Speech Processing |
| topic | Audio and Speech Processing Artificial Intelligence Computation and Language Sound |
| url | https://arxiv.org/abs/2408.05758 |