Eliminating stability hallucinations in llm-based tts models via attention guidance
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908830814699520 |
|---|---|
| author | Wang, ShiMing Du, ZhiHao Xiang, Yang Zhao, TianYu Zhao, Han Chen, Qian Li, XianGang Guo, HanJie Ling, ZhenHua |
| author_facet | Wang, ShiMing Du, ZhiHao Xiang, Yang Zhao, TianYu Zhao, Han Chen, Qian Li, XianGang Guo, HanJie Ling, ZhenHua |
| contents | This paper focuses on resolving stability hallucinations (e.g., repetitive or omitted speech) in LLM-based Text-to-Speech (TTS) models by improving and leveraging the attention mechanism. First, we analyzed the alignment mechanism between text tokens and speech tokens in LLMs. We then proposed a metric termed the Optimal Alignment Score (OAS), which employs the Viterbi algorithm to evaluate text-speech alignment quality. Subsequently, OAS was integrated into the training of CosyVoice2 to assist LLMs in learning continuous, stable alignment. Additionally, the pre-trained attention value is employed to guide the training of the student CosyVoice2 via chain-of-thought (CoT), which further reduces stability hallucinations in synthesized speech. Experiments on the Seed-TTS-Eval and CV3-Eval test sets demonstrate that the proposed methods can effectively reduce the stability hallucinations of CosyVoice2 without introducing additional negative effects. The appendix is available at https://wsmzzz.github.io/llm_attn. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_19852 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Eliminating stability hallucinations in llm-based tts models via attention guidance Wang, ShiMing Du, ZhiHao Xiang, Yang Zhao, TianYu Zhao, Han Chen, Qian Li, XianGang Guo, HanJie Ling, ZhenHua Sound Artificial Intelligence This paper focuses on resolving stability hallucinations (e.g., repetitive or omitted speech) in LLM-based Text-to-Speech (TTS) models by improving and leveraging the attention mechanism. First, we analyzed the alignment mechanism between text tokens and speech tokens in LLMs. We then proposed a metric termed the Optimal Alignment Score (OAS), which employs the Viterbi algorithm to evaluate text-speech alignment quality. Subsequently, OAS was integrated into the training of CosyVoice2 to assist LLMs in learning continuous, stable alignment. Additionally, the pre-trained attention value is employed to guide the training of the student CosyVoice2 via chain-of-thought (CoT), which further reduces stability hallucinations in synthesized speech. Experiments on the Seed-TTS-Eval and CV3-Eval test sets demonstrate that the proposed methods can effectively reduce the stability hallucinations of CosyVoice2 without introducing additional negative effects. The appendix is available at https://wsmzzz.github.io/llm_attn. |
| title | Eliminating stability hallucinations in llm-based tts models via attention guidance |
| topic | Sound Artificial Intelligence |
| url | https://arxiv.org/abs/2509.19852 |