Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2401.01885 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913184780124160 |
|---|---|
| author | Ng, Evonne Romero, Javier Bagautdinov, Timur Bai, Shaojie Darrell, Trevor Kanazawa, Angjoo Richard, Alexander |
| author_facet | Ng, Evonne Romero, Javier Bagautdinov, Timur Bai, Shaojie Darrell, Trevor Kanazawa, Angjoo Richard, Alexander |
| contents | We present a framework for generating full-bodied photorealistic avatars that gesture according to the conversational dynamics of a dyadic interaction. Given speech audio, we output multiple possibilities of gestural motion for an individual, including face, body, and hands. The key behind our method is in combining the benefits of sample diversity from vector quantization with the high-frequency details obtained through diffusion to generate more dynamic, expressive motion. We visualize the generated motion using highly photorealistic avatars that can express crucial nuances in gestures (e.g. sneers and smirks). To facilitate this line of research, we introduce a first-of-its-kind multi-view conversational dataset that allows for photorealistic reconstruction. Experiments show our model generates appropriate and diverse gestures, outperforming both diffusion- and VQ-only methods. Furthermore, our perceptual evaluation highlights the importance of photorealism (vs. meshes) in accurately assessing subtle motion details in conversational gestures. Code and dataset available online. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_01885 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations Ng, Evonne Romero, Javier Bagautdinov, Timur Bai, Shaojie Darrell, Trevor Kanazawa, Angjoo Richard, Alexander Computer Vision and Pattern Recognition We present a framework for generating full-bodied photorealistic avatars that gesture according to the conversational dynamics of a dyadic interaction. Given speech audio, we output multiple possibilities of gestural motion for an individual, including face, body, and hands. The key behind our method is in combining the benefits of sample diversity from vector quantization with the high-frequency details obtained through diffusion to generate more dynamic, expressive motion. We visualize the generated motion using highly photorealistic avatars that can express crucial nuances in gestures (e.g. sneers and smirks). To facilitate this line of research, we introduce a first-of-its-kind multi-view conversational dataset that allows for photorealistic reconstruction. Experiments show our model generates appropriate and diverse gestures, outperforming both diffusion- and VQ-only methods. Furthermore, our perceptual evaluation highlights the importance of photorealism (vs. meshes) in accurately assessing subtle motion details in conversational gestures. Code and dataset available online. |
| title | From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2401.01885 |