EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913987163062272 |
|---|---|
| author | Yang, Guanrou Yang, Chen Chen, Qian Ma, Ziyang Chen, Wenxi Wang, Wen Wang, Tianrui Yang, Yifan Niu, Zhikang Liu, Wenrui Yu, Fan Du, Zhihao Gao, Zhifu Zhang, ShiLiang Chen, Xie |
| author_facet | Yang, Guanrou Yang, Chen Chen, Qian Ma, Ziyang Chen, Wenxi Wang, Wen Wang, Tianrui Yang, Yifan Niu, Zhikang Liu, Wenrui Yu, Fan Du, Zhihao Gao, Zhifu Zhang, ShiLiang Chen, Xie |
| contents | Human speech goes beyond the mere transfer of information; it is a profound exchange of emotions and a connection between individuals. While Text-to-Speech (TTS) models have made huge progress, they still face challenges in controlling the emotional expression in the generated speech. In this work, we propose EmoVoice, a novel emotion-controllable TTS model that exploits large language models (LLMs) to enable fine-grained freestyle natural language emotion control, and a phoneme boost variant design that makes the model output phoneme tokens and audio tokens in parallel to enhance content consistency, inspired by chain-of-thought (CoT) and chain-of-modality (CoM) techniques. Besides, we introduce EmoVoice-DB, a high-quality 40-hour English emotion dataset featuring expressive speech and fine-grained emotion labels with natural language descriptions. EmoVoice achieves state-of-the-art performance on the English EmoVoice-DB test set using only synthetic training data, and on the Chinese Secap test set using our in-house data. We further investigate the reliability of existing emotion evaluation metrics and their alignment with human perceptual preferences, and explore using SOTA multimodal LLMs GPT-4o-audio and Gemini to assess emotional speech. Dataset, code, checkpoints, and demo samples are available at https://github.com/yanghaha0908/EmoVoice. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_12867 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting Yang, Guanrou Yang, Chen Chen, Qian Ma, Ziyang Chen, Wenxi Wang, Wen Wang, Tianrui Yang, Yifan Niu, Zhikang Liu, Wenrui Yu, Fan Du, Zhihao Gao, Zhifu Zhang, ShiLiang Chen, Xie Audio and Speech Processing Artificial Intelligence Computation and Language Human speech goes beyond the mere transfer of information; it is a profound exchange of emotions and a connection between individuals. While Text-to-Speech (TTS) models have made huge progress, they still face challenges in controlling the emotional expression in the generated speech. In this work, we propose EmoVoice, a novel emotion-controllable TTS model that exploits large language models (LLMs) to enable fine-grained freestyle natural language emotion control, and a phoneme boost variant design that makes the model output phoneme tokens and audio tokens in parallel to enhance content consistency, inspired by chain-of-thought (CoT) and chain-of-modality (CoM) techniques. Besides, we introduce EmoVoice-DB, a high-quality 40-hour English emotion dataset featuring expressive speech and fine-grained emotion labels with natural language descriptions. EmoVoice achieves state-of-the-art performance on the English EmoVoice-DB test set using only synthetic training data, and on the Chinese Secap test set using our in-house data. We further investigate the reliability of existing emotion evaluation metrics and their alignment with human perceptual preferences, and explore using SOTA multimodal LLMs GPT-4o-audio and Gemini to assess emotional speech. Dataset, code, checkpoints, and demo samples are available at https://github.com/yanghaha0908/EmoVoice. |
| title | EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting |
| topic | Audio and Speech Processing Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2504.12867 |