DEGAS: Detailed Expressions on Full-Body Gaussian Avatars
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917916610396160 |
|---|---|
| author | Shao, Zhijing Wang, Duotun Tian, Qing-Yao Yang, Yao-Dong Meng, Hengyu Cai, Zeyu Dong, Bo Zhang, Yu Zhang, Kang Wang, Zeyu |
| author_facet | Shao, Zhijing Wang, Duotun Tian, Qing-Yao Yang, Yao-Dong Meng, Hengyu Cai, Zeyu Dong, Bo Zhang, Yu Zhang, Kang Wang, Zeyu |
| contents | Although neural rendering has made significant advances in creating lifelike, animatable full-body and head avatars, incorporating detailed expressions into full-body avatars remains largely unexplored. We present DEGAS, the first 3D Gaussian Splatting (3DGS)-based modeling method for full-body avatars with rich facial expressions. Trained on multiview videos of a given subject, our method learns a conditional variational autoencoder that takes both the body motion and facial expression as driving signals to generate Gaussian maps in the UV layout. To drive the facial expressions, instead of the commonly used 3D Morphable Models (3DMMs) in 3D head avatars, we propose to adopt the expression latent space trained solely on 2D portrait images, bridging the gap between 2D talking faces and 3D avatars. Leveraging the rendering capability of 3DGS and the rich expressiveness of the expression latent space, the learned avatars can be reenacted to reproduce photorealistic rendering images with subtle and accurate facial expressions. Experiments on an existing dataset and our newly proposed dataset of full-body talking avatars demonstrate the efficacy of our method. We also propose an audio-driven extension of our method with the help of 2D talking faces, opening new possibilities for interactive AI agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_10588 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DEGAS: Detailed Expressions on Full-Body Gaussian Avatars Shao, Zhijing Wang, Duotun Tian, Qing-Yao Yang, Yao-Dong Meng, Hengyu Cai, Zeyu Dong, Bo Zhang, Yu Zhang, Kang Wang, Zeyu Computer Vision and Pattern Recognition Graphics Although neural rendering has made significant advances in creating lifelike, animatable full-body and head avatars, incorporating detailed expressions into full-body avatars remains largely unexplored. We present DEGAS, the first 3D Gaussian Splatting (3DGS)-based modeling method for full-body avatars with rich facial expressions. Trained on multiview videos of a given subject, our method learns a conditional variational autoencoder that takes both the body motion and facial expression as driving signals to generate Gaussian maps in the UV layout. To drive the facial expressions, instead of the commonly used 3D Morphable Models (3DMMs) in 3D head avatars, we propose to adopt the expression latent space trained solely on 2D portrait images, bridging the gap between 2D talking faces and 3D avatars. Leveraging the rendering capability of 3DGS and the rich expressiveness of the expression latent space, the learned avatars can be reenacted to reproduce photorealistic rendering images with subtle and accurate facial expressions. Experiments on an existing dataset and our newly proposed dataset of full-body talking avatars demonstrate the efficacy of our method. We also propose an audio-driven extension of our method with the help of 2D talking faces, opening new possibilities for interactive AI agents. |
| title | DEGAS: Detailed Expressions on Full-Body Gaussian Avatars |
| topic | Computer Vision and Pattern Recognition Graphics |
| url | https://arxiv.org/abs/2408.10588 |