TalkinNeRF: Animatable Neural Fields for Full-Body Talking Humans
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917786611089408 |
|---|---|
| author | Chatziagapi, Aggelina Chaudhuri, Bindita Kumar, Amit Ranjan, Rakesh Samaras, Dimitris Sarafianos, Nikolaos |
| author_facet | Chatziagapi, Aggelina Chaudhuri, Bindita Kumar, Amit Ranjan, Rakesh Samaras, Dimitris Sarafianos, Nikolaos |
| contents | We introduce a novel framework that learns a dynamic neural radiance field (NeRF) for full-body talking humans from monocular videos. Prior work represents only the body pose or the face. However, humans communicate with their full body, combining body pose, hand gestures, as well as facial expressions. In this work, we propose TalkinNeRF, a unified NeRF-based network that represents the holistic 4D human motion. Given a monocular video of a subject, we learn corresponding modules for the body, face, and hands, that are combined together to generate the final result. To capture complex finger articulation, we learn an additional deformation field for the hands. Our multi-identity representation enables simultaneous training for multiple subjects, as well as robust animation under completely unseen poses. It can also generalize to novel identities, given only a short video as input. We demonstrate state-of-the-art performance for animating full-body talking humans, with fine-grained hand articulation and facial expressions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_16666 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TalkinNeRF: Animatable Neural Fields for Full-Body Talking Humans Chatziagapi, Aggelina Chaudhuri, Bindita Kumar, Amit Ranjan, Rakesh Samaras, Dimitris Sarafianos, Nikolaos Computer Vision and Pattern Recognition We introduce a novel framework that learns a dynamic neural radiance field (NeRF) for full-body talking humans from monocular videos. Prior work represents only the body pose or the face. However, humans communicate with their full body, combining body pose, hand gestures, as well as facial expressions. In this work, we propose TalkinNeRF, a unified NeRF-based network that represents the holistic 4D human motion. Given a monocular video of a subject, we learn corresponding modules for the body, face, and hands, that are combined together to generate the final result. To capture complex finger articulation, we learn an additional deformation field for the hands. Our multi-identity representation enables simultaneous training for multiple subjects, as well as robust animation under completely unseen poses. It can also generalize to novel identities, given only a short video as input. We demonstrate state-of-the-art performance for animating full-body talking humans, with fine-grained hand articulation and facial expressions. |
| title | TalkinNeRF: Animatable Neural Fields for Full-Body Talking Humans |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2409.16666 |