Faces that Speak: Jointly Synthesising Talking Face and Speech from Text
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914799016738816 |
|---|---|
| author | Jang, Youngjoon Kim, Ji-Hoon Ahn, Junseok Kwak, Doyeop Yang, Hong-Sun Ju, Yoon-Cheol Kim, Il-Hwan Kim, Byeong-Yeol Chung, Joon Son |
| author_facet | Jang, Youngjoon Kim, Ji-Hoon Ahn, Junseok Kwak, Doyeop Yang, Hong-Sun Ju, Yoon-Cheol Kim, Il-Hwan Kim, Byeong-Yeol Chung, Joon Son |
| contents | The goal of this work is to simultaneously generate natural talking faces and speech outputs from text. We achieve this by integrating Talking Face Generation (TFG) and Text-to-Speech (TTS) systems into a unified framework. We address the main challenges of each task: (1) generating a range of head poses representative of real-world scenarios, and (2) ensuring voice consistency despite variations in facial motion for the same identity. To tackle these issues, we introduce a motion sampler based on conditional flow matching, which is capable of high-quality motion code generation in an efficient way. Moreover, we introduce a novel conditioning method for the TTS system, which utilises motion-removed features from the TFG model to yield uniform speech outputs. Our extensive experiments demonstrate that our method effectively creates natural-looking talking faces and speech that accurately match the input text. To our knowledge, this is the first effort to build a multimodal synthesis system that can generalise to unseen identities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_10272 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Faces that Speak: Jointly Synthesising Talking Face and Speech from Text Jang, Youngjoon Kim, Ji-Hoon Ahn, Junseok Kwak, Doyeop Yang, Hong-Sun Ju, Yoon-Cheol Kim, Il-Hwan Kim, Byeong-Yeol Chung, Joon Son Computer Vision and Pattern Recognition Artificial Intelligence Sound Audio and Speech Processing Image and Video Processing The goal of this work is to simultaneously generate natural talking faces and speech outputs from text. We achieve this by integrating Talking Face Generation (TFG) and Text-to-Speech (TTS) systems into a unified framework. We address the main challenges of each task: (1) generating a range of head poses representative of real-world scenarios, and (2) ensuring voice consistency despite variations in facial motion for the same identity. To tackle these issues, we introduce a motion sampler based on conditional flow matching, which is capable of high-quality motion code generation in an efficient way. Moreover, we introduce a novel conditioning method for the TTS system, which utilises motion-removed features from the TFG model to yield uniform speech outputs. Our extensive experiments demonstrate that our method effectively creates natural-looking talking faces and speech that accurately match the input text. To our knowledge, this is the first effort to build a multimodal synthesis system that can generalise to unseen identities. |
| title | Faces that Speak: Jointly Synthesising Talking Face and Speech from Text |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Sound Audio and Speech Processing Image and Video Processing |
| url | https://arxiv.org/abs/2405.10272 |