TalkCLIP: Talking Head Generation with Text-Guided Expressive Speaking Styles
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909283030925312 |
|---|---|
| author | Ma, Yifeng Wang, Suzhen Ding, Yu Ma, Bowen Lv, Tangjie Fan, Changjie Hu, Zhipeng Deng, Zhidong Yu, Xin |
| author_facet | Ma, Yifeng Wang, Suzhen Ding, Yu Ma, Bowen Lv, Tangjie Fan, Changjie Hu, Zhipeng Deng, Zhidong Yu, Xin |
| contents | Audio-driven talking head generation has drawn growing attention. To produce talking head videos with desired facial expressions, previous methods rely on extra reference videos to provide expression information, which may be difficult to find and hence limits their usage. In this work, we propose TalkCLIP, a framework that can generate talking heads where the expressions are specified by natural language, hence allowing for specifying expressions more conveniently. To model the mapping from text to expressions, we first construct a text-video paired talking head dataset where each video has diverse text descriptions that depict both coarse-grained emotions and fine-grained facial movements. Leveraging the proposed dataset, we introduce a CLIP-based style encoder that projects natural language-based descriptions to the representations of expressions. TalkCLIP can even infer expressions for descriptions unseen during training. TalkCLIP can also use text to modulate expression intensity and edit expressions. Extensive experiments demonstrate that TalkCLIP achieves the advanced capability of generating photo-realistic talking heads with vivid facial expressions guided by text descriptions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2304_00334 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | TalkCLIP: Talking Head Generation with Text-Guided Expressive Speaking Styles Ma, Yifeng Wang, Suzhen Ding, Yu Ma, Bowen Lv, Tangjie Fan, Changjie Hu, Zhipeng Deng, Zhidong Yu, Xin Computer Vision and Pattern Recognition Audio-driven talking head generation has drawn growing attention. To produce talking head videos with desired facial expressions, previous methods rely on extra reference videos to provide expression information, which may be difficult to find and hence limits their usage. In this work, we propose TalkCLIP, a framework that can generate talking heads where the expressions are specified by natural language, hence allowing for specifying expressions more conveniently. To model the mapping from text to expressions, we first construct a text-video paired talking head dataset where each video has diverse text descriptions that depict both coarse-grained emotions and fine-grained facial movements. Leveraging the proposed dataset, we introduce a CLIP-based style encoder that projects natural language-based descriptions to the representations of expressions. TalkCLIP can even infer expressions for descriptions unseen during training. TalkCLIP can also use text to modulate expression intensity and edit expressions. Extensive experiments demonstrate that TalkCLIP achieves the advanced capability of generating photo-realistic talking heads with vivid facial expressions guided by text descriptions. |
| title | TalkCLIP: Talking Head Generation with Text-Guided Expressive Speaking Styles |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2304.00334 |