SemaVoice: Semantic-Aware Continuous Autoregressive Speech Synthesis
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916018617581568 |
|---|---|
| author | Wang, Huimeng Lu, Hui Deng, Jiajun Xu, Haoning Chen, Youjun Chen, Xueyuan Li, Zhaoqing Peng, Shuhai Kang, Shiyin Liu, Xunying |
| author_facet | Wang, Huimeng Lu, Hui Deng, Jiajun Xu, Haoning Chen, Youjun Chen, Xueyuan Li, Zhaoqing Peng, Shuhai Kang, Shiyin Liu, Xunying |
| contents | Continuous autoregressive speech synthesis has recently emerged as a promising direction for zero-shot text-to-speech (TTS). However, existing methods still suffer from a fundamental mismatch between semantic-prosodic modeling and reconstruction-driven continuous speech representations. This mismatch causes TTS models to focus excessively on low-level acoustic textures at the expense of high-level semantic coherence, further exacerbating error accumulation in autoregressive generation. To address this challenge, we propose SemaVoice, a semantic-aware continuous autoregressive framework for high-fidelity zero-shot TTS. SemaVoice introduces a Speech Foundation Model (SFM) guided alignment mechanism that refines continuous speech representations to better capture both local semantic consistency and global structural relationships. These representations condition a patch-wise diffusion head within the autoregressive framework for high-quality speech synthesis. Experimental results on the Seed-TTS benchmark show that SemaVoice achieves an English WER of 1.71\% and remains highly competitive with state-of-the-art open-source systems in both objective and subjective evaluations. The effectiveness of SFM guided alignment is further confirmed by significant improvements under varying representation granularities with a fixed information-rate constraint. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_16964 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SemaVoice: Semantic-Aware Continuous Autoregressive Speech Synthesis Wang, Huimeng Lu, Hui Deng, Jiajun Xu, Haoning Chen, Youjun Chen, Xueyuan Li, Zhaoqing Peng, Shuhai Kang, Shiyin Liu, Xunying Audio and Speech Processing Continuous autoregressive speech synthesis has recently emerged as a promising direction for zero-shot text-to-speech (TTS). However, existing methods still suffer from a fundamental mismatch between semantic-prosodic modeling and reconstruction-driven continuous speech representations. This mismatch causes TTS models to focus excessively on low-level acoustic textures at the expense of high-level semantic coherence, further exacerbating error accumulation in autoregressive generation. To address this challenge, we propose SemaVoice, a semantic-aware continuous autoregressive framework for high-fidelity zero-shot TTS. SemaVoice introduces a Speech Foundation Model (SFM) guided alignment mechanism that refines continuous speech representations to better capture both local semantic consistency and global structural relationships. These representations condition a patch-wise diffusion head within the autoregressive framework for high-quality speech synthesis. Experimental results on the Seed-TTS benchmark show that SemaVoice achieves an English WER of 1.71\% and remains highly competitive with state-of-the-art open-source systems in both objective and subjective evaluations. The effectiveness of SFM guided alignment is further confirmed by significant improvements under varying representation granularities with a fixed information-rate constraint. |
| title | SemaVoice: Semantic-Aware Continuous Autoregressive Speech Synthesis |
| topic | Audio and Speech Processing |
| url | https://arxiv.org/abs/2605.16964 |