SpeechAccentLLM: A Unified Framework for Foreign Accent Conversion and Text to Speech
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912471337402368 |
|---|---|
| author | Cheng, Zhuangfei Zhang, Guangyan Tu, Zehai Song, Yangyang Mao, Shuiyang Jiao, Xiaoqi Li, Jingyu Guo, Yiwen Wu, Jiasong |
| author_facet | Cheng, Zhuangfei Zhang, Guangyan Tu, Zehai Song, Yangyang Mao, Shuiyang Jiao, Xiaoqi Li, Jingyu Guo, Yiwen Wu, Jiasong |
| contents | Foreign accent conversion (FAC) in speech processing remains a challenging task. Building on the remarkable success of large language models (LLMs) in Text-to-Speech (TTS) tasks, this study investigates the adaptation of LLM-based techniques for FAC, which we term SpeechAccentLLM. At the core of this framework, we introduce SpeechCodeVAE, the first model to integrate connectionist temporal classification (CTC) directly into codebook discretization for speech content tokenization. This novel architecture generates tokens with a unique "locality" property, as validated by experiments demonstrating optimal trade-offs among content faithfulness, temporal coherence, and structural recoverability. Then, to address data scarcity for the FAC module, we adopted a multitask learning strategy that jointly trains the FAC and TTS modules. Beyond mitigating data limitations, this approach yielded accelerated convergence and superior speech quality compared to standalone FAC training. Moreover, leveraging the salient properties of our discrete speech representations, we introduce SpeechRestorer, a postprocessing architecture designed to refine LLM-generated outputs. This module effectively mitigates stochastic errors prevalent in LLM inference pipelines while enhancing prosodic continuity, as validated by ablation experiments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_01348 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SpeechAccentLLM: A Unified Framework for Foreign Accent Conversion and Text to Speech Cheng, Zhuangfei Zhang, Guangyan Tu, Zehai Song, Yangyang Mao, Shuiyang Jiao, Xiaoqi Li, Jingyu Guo, Yiwen Wu, Jiasong Audio and Speech Processing Sound I.2.7 Foreign accent conversion (FAC) in speech processing remains a challenging task. Building on the remarkable success of large language models (LLMs) in Text-to-Speech (TTS) tasks, this study investigates the adaptation of LLM-based techniques for FAC, which we term SpeechAccentLLM. At the core of this framework, we introduce SpeechCodeVAE, the first model to integrate connectionist temporal classification (CTC) directly into codebook discretization for speech content tokenization. This novel architecture generates tokens with a unique "locality" property, as validated by experiments demonstrating optimal trade-offs among content faithfulness, temporal coherence, and structural recoverability. Then, to address data scarcity for the FAC module, we adopted a multitask learning strategy that jointly trains the FAC and TTS modules. Beyond mitigating data limitations, this approach yielded accelerated convergence and superior speech quality compared to standalone FAC training. Moreover, leveraging the salient properties of our discrete speech representations, we introduce SpeechRestorer, a postprocessing architecture designed to refine LLM-generated outputs. This module effectively mitigates stochastic errors prevalent in LLM inference pipelines while enhancing prosodic continuity, as validated by ablation experiments. |
| title | SpeechAccentLLM: A Unified Framework for Foreign Accent Conversion and Text to Speech |
| topic | Audio and Speech Processing Sound I.2.7 |
| url | https://arxiv.org/abs/2507.01348 |