Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2601.18451 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914280900657152 |
|---|---|
| author | Sha, Xuanmeng Zhang, Liyun Mashita, Tomohiro Chiba, Naoya Uranishi, Yuki |
| author_facet | Sha, Xuanmeng Zhang, Liyun Mashita, Tomohiro Chiba, Naoya Uranishi, Yuki |
| contents | Generating holistic co-speech gestures that integrate full-body motion with facial expressions suffers from semantically incoherent coordination on body motion and spatially unstable meaningless movements due to existing part-decomposed or frame-level regression methods, We introduce 3DGesPolicy, a novel action-based framework that reformulates holistic gesture generation as a continuous trajectory control problem through diffusion policy from robotics. By modeling frame-to-frame variations as unified holistic actions, our method effectively learns inter-frame holistic gesture motion patterns and ensures both spatially and semantically coherent movement trajectories that adhere to realistic motion manifolds. To further bridge the gap in expressive alignment, we propose a Gesture-Audio-Phoneme (GAP) fusion module that can deeply integrate and refine multi-modal signals, ensuring structured and fine-grained alignment between speech semantics, body motion, and facial expressions. Extensive quantitative and qualitative experiments on the BEAT2 dataset demonstrate the effectiveness of our 3DGesPolicy across other state-of-the-art methods in generating natural, expressive, and highly speech-aligned holistic gestures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_18451 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | 3DGesPolicy: Phoneme-Aware Holistic Co-Speech Gesture Generation Based on Action Control Sha, Xuanmeng Zhang, Liyun Mashita, Tomohiro Chiba, Naoya Uranishi, Yuki Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Multimedia Sound I.3.7; I.2.10 Generating holistic co-speech gestures that integrate full-body motion with facial expressions suffers from semantically incoherent coordination on body motion and spatially unstable meaningless movements due to existing part-decomposed or frame-level regression methods, We introduce 3DGesPolicy, a novel action-based framework that reformulates holistic gesture generation as a continuous trajectory control problem through diffusion policy from robotics. By modeling frame-to-frame variations as unified holistic actions, our method effectively learns inter-frame holistic gesture motion patterns and ensures both spatially and semantically coherent movement trajectories that adhere to realistic motion manifolds. To further bridge the gap in expressive alignment, we propose a Gesture-Audio-Phoneme (GAP) fusion module that can deeply integrate and refine multi-modal signals, ensuring structured and fine-grained alignment between speech semantics, body motion, and facial expressions. Extensive quantitative and qualitative experiments on the BEAT2 dataset demonstrate the effectiveness of our 3DGesPolicy across other state-of-the-art methods in generating natural, expressive, and highly speech-aligned holistic gestures. |
| title | 3DGesPolicy: Phoneme-Aware Holistic Co-Speech Gesture Generation Based on Action Control |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Multimedia Sound I.3.7; I.2.10 |
| url | https://arxiv.org/abs/2601.18451 |