Controlling Emotion in Text-to-Speech with Natural Language Prompts
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929382520520704 |
|---|---|
| author | Bott, Thomas Lux, Florian Vu, Ngoc Thang |
| author_facet | Bott, Thomas Lux, Florian Vu, Ngoc Thang |
| contents | In recent years, prompting has quickly become one of the standard ways of steering the outputs of generative machine learning models, due to its intuitive use of natural language. In this work, we propose a system conditioned on embeddings derived from an emotionally rich text that serves as prompt. Thereby, a joint representation of speaker and prompt embeddings is integrated at several points within a transformer-based architecture. Our approach is trained on merged emotional speech and text datasets and varies prompts in each training iteration to increase the generalization capabilities of the model. Objective and subjective evaluation results demonstrate the ability of the conditioned synthesis system to accurately transfer the emotions present in a prompt to speech. At the same time, precise tractability of speaker identities as well as overall high speech quality and intelligibility are maintained. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_06406 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Controlling Emotion in Text-to-Speech with Natural Language Prompts Bott, Thomas Lux, Florian Vu, Ngoc Thang Computation and Language Sound Audio and Speech Processing In recent years, prompting has quickly become one of the standard ways of steering the outputs of generative machine learning models, due to its intuitive use of natural language. In this work, we propose a system conditioned on embeddings derived from an emotionally rich text that serves as prompt. Thereby, a joint representation of speaker and prompt embeddings is integrated at several points within a transformer-based architecture. Our approach is trained on merged emotional speech and text datasets and varies prompts in each training iteration to increase the generalization capabilities of the model. Objective and subjective evaluation results demonstrate the ability of the conditioned synthesis system to accurately transfer the emotions present in a prompt to speech. At the same time, precise tractability of speaker identities as well as overall high speech quality and intelligibility are maintained. |
| title | Controlling Emotion in Text-to-Speech with Natural Language Prompts |
| topic | Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2406.06406 |