Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.16563 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910869229666304 |
|---|---|
| author | Zhang, Xiangyue Li, Jianfang Zhang, Jiaxu Dang, Ziqiang Ren, Jianqiang Bo, Liefeng Tu, Zhigang |
| author_facet | Zhang, Xiangyue Li, Jianfang Zhang, Jiaxu Dang, Ziqiang Ren, Jianqiang Bo, Liefeng Tu, Zhigang |
| contents | A good co-speech motion generation cannot be achieved without a careful integration of common rhythmic motion and rare yet essential semantic motion. In this work, we propose SemTalk for holistic co-speech motion generation with frame-level semantic emphasis. Our key insight is to separately learn base motions and sparse motions, and then adaptively fuse them. In particular, coarse2fine cross-attention module and rhythmic consistency learning are explored to establish rhythm-related base motion, ensuring a coherent foundation that synchronizes gestures with the speech rhythm. Subsequently, semantic emphasis learning is designed to generate semantic-aware sparse motion, focusing on frame-level semantic cues. Finally, to integrate sparse motion into the base motion and generate semantic-emphasized co-speech gestures, we further leverage a learned semantic score for adaptive synthesis. Qualitative and quantitative comparisons on two public datasets demonstrate that our method outperforms the state-of-the-art, delivering high-quality co-speech motion with enhanced semantic richness over a stable base motion. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_16563 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasis Zhang, Xiangyue Li, Jianfang Zhang, Jiaxu Dang, Ziqiang Ren, Jianqiang Bo, Liefeng Tu, Zhigang Computer Vision and Pattern Recognition A good co-speech motion generation cannot be achieved without a careful integration of common rhythmic motion and rare yet essential semantic motion. In this work, we propose SemTalk for holistic co-speech motion generation with frame-level semantic emphasis. Our key insight is to separately learn base motions and sparse motions, and then adaptively fuse them. In particular, coarse2fine cross-attention module and rhythmic consistency learning are explored to establish rhythm-related base motion, ensuring a coherent foundation that synchronizes gestures with the speech rhythm. Subsequently, semantic emphasis learning is designed to generate semantic-aware sparse motion, focusing on frame-level semantic cues. Finally, to integrate sparse motion into the base motion and generate semantic-emphasized co-speech gestures, we further leverage a learned semantic score for adaptive synthesis. Qualitative and quantitative comparisons on two public datasets demonstrate that our method outperforms the state-of-the-art, delivering high-quality co-speech motion with enhanced semantic richness over a stable base motion. |
| title | SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasis |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.16563 |