Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917776723017728 |
|---|---|
| author | Gao, Xiaoxue Zhang, Chen Chen, Yiming Zhang, Huayun Chen, Nancy F. |
| author_facet | Gao, Xiaoxue Zhang, Chen Chen, Yiming Zhang, Huayun Chen, Nancy F. |
| contents | Current emotional text-to-speech (TTS) models predominantly conduct supervised training to learn the conversion from text and desired emotion to its emotional speech, focusing on a single emotion per text-speech pair. These models only learn the correct emotional outputs without fully comprehending other emotion characteristics, which limits their capabilities of capturing the nuances between different emotions. We propose a controllable Emo-DPO approach, which employs direct preference optimization to differentiate subtle emotional nuances between emotions through optimizing towards preferred emotions over less preferred emotional ones. Instead of relying on traditional neural architectures used in existing emotional TTS models, we propose utilizing the emotion-aware LLM-TTS neural architecture to leverage LLMs' in-context learning and instruction-following capabilities. Comprehensive experiments confirm that our proposed method outperforms the existing baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_10157 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimization Gao, Xiaoxue Zhang, Chen Chen, Yiming Zhang, Huayun Chen, Nancy F. Audio and Speech Processing Sound Signal Processing Current emotional text-to-speech (TTS) models predominantly conduct supervised training to learn the conversion from text and desired emotion to its emotional speech, focusing on a single emotion per text-speech pair. These models only learn the correct emotional outputs without fully comprehending other emotion characteristics, which limits their capabilities of capturing the nuances between different emotions. We propose a controllable Emo-DPO approach, which employs direct preference optimization to differentiate subtle emotional nuances between emotions through optimizing towards preferred emotions over less preferred emotional ones. Instead of relying on traditional neural architectures used in existing emotional TTS models, we propose utilizing the emotion-aware LLM-TTS neural architecture to leverage LLMs' in-context learning and instruction-following capabilities. Comprehensive experiments confirm that our proposed method outperforms the existing baselines. |
| title | Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimization |
| topic | Audio and Speech Processing Sound Signal Processing |
| url | https://arxiv.org/abs/2409.10157 |