One Shot, One Talk: Whole-body Talking Avatar from a Single Image
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929610360356864 |
|---|---|
| author | Xiang, Jun Guo, Yudong Hu, Leipeng Guo, Boyang Yuan, Yancheng Zhang, Juyong |
| author_facet | Xiang, Jun Guo, Yudong Hu, Leipeng Guo, Boyang Yuan, Yancheng Zhang, Juyong |
| contents | Building realistic and animatable avatars still requires minutes of multi-view or monocular self-rotating videos, and most methods lack precise control over gestures and expressions. To push this boundary, we address the challenge of constructing a whole-body talking avatar from a single image. We propose a novel pipeline that tackles two critical issues: 1) complex dynamic modeling and 2) generalization to novel gestures and expressions. To achieve seamless generalization, we leverage recent pose-guided image-to-video diffusion models to generate imperfect video frames as pseudo-labels. To overcome the dynamic modeling challenge posed by inconsistent and noisy pseudo-videos, we introduce a tightly coupled 3DGS-mesh hybrid avatar representation and apply several key regularizations to mitigate inconsistencies caused by imperfect labels. Extensive experiments on diverse subjects demonstrate that our method enables the creation of a photorealistic, precisely animatable, and expressive whole-body talking avatar from just a single image. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_01106 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | One Shot, One Talk: Whole-body Talking Avatar from a Single Image Xiang, Jun Guo, Yudong Hu, Leipeng Guo, Boyang Yuan, Yancheng Zhang, Juyong Computer Vision and Pattern Recognition Graphics Building realistic and animatable avatars still requires minutes of multi-view or monocular self-rotating videos, and most methods lack precise control over gestures and expressions. To push this boundary, we address the challenge of constructing a whole-body talking avatar from a single image. We propose a novel pipeline that tackles two critical issues: 1) complex dynamic modeling and 2) generalization to novel gestures and expressions. To achieve seamless generalization, we leverage recent pose-guided image-to-video diffusion models to generate imperfect video frames as pseudo-labels. To overcome the dynamic modeling challenge posed by inconsistent and noisy pseudo-videos, we introduce a tightly coupled 3DGS-mesh hybrid avatar representation and apply several key regularizations to mitigate inconsistencies caused by imperfect labels. Extensive experiments on diverse subjects demonstrate that our method enables the creation of a photorealistic, precisely animatable, and expressive whole-body talking avatar from just a single image. |
| title | One Shot, One Talk: Whole-body Talking Avatar from a Single Image |
| topic | Computer Vision and Pattern Recognition Graphics |
| url | https://arxiv.org/abs/2412.01106 |