Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.17674 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913519686909952 |
|---|---|
| author | Yang, Huan Chen, Jiahui Ding, Chaofan Shi, Runhua Xiong, Siyu Hong, Qingqi Mo, Xiaoqi Di, Xinhan |
| author_facet | Yang, Huan Chen, Jiahui Ding, Chaofan Shi, Runhua Xiong, Siyu Hong, Qingqi Mo, Xiaoqi Di, Xinhan |
| contents | Gestures are pivotal in enhancing co-speech communication. While recent works have mostly focused on point-level motion transformation or fully supervised motion representations through data-driven approaches, we explore the representation of gestures in co-speech, with a focus on self-supervised representation and pixel-level motion deviation, utilizing a diffusion model which incorporates latent motion features. Our approach leverages self-supervised deviation in latent representation to facilitate hand gestures generation, which are crucial for generating realistic gesture videos. Results of our first experiment demonstrate that our method enhances the quality of generated videos, with an improvement from 2.7 to 4.5% for FGD, DIV, and FVD, and 8.1% for PSNR, 2.5% for SSIM over the current state-of-the-art methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_17674 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Self-Supervised Learning of Deviation in Latent Representation for Co-speech Gesture Video Generation Yang, Huan Chen, Jiahui Ding, Chaofan Shi, Runhua Xiong, Siyu Hong, Qingqi Mo, Xiaoqi Di, Xinhan Computer Vision and Pattern Recognition Gestures are pivotal in enhancing co-speech communication. While recent works have mostly focused on point-level motion transformation or fully supervised motion representations through data-driven approaches, we explore the representation of gestures in co-speech, with a focus on self-supervised representation and pixel-level motion deviation, utilizing a diffusion model which incorporates latent motion features. Our approach leverages self-supervised deviation in latent representation to facilitate hand gestures generation, which are crucial for generating realistic gesture videos. Results of our first experiment demonstrate that our method enhances the quality of generated videos, with an improvement from 2.7 to 4.5% for FGD, DIV, and FVD, and 8.1% for PSNR, 2.5% for SSIM over the current state-of-the-art methods. |
| title | Self-Supervised Learning of Deviation in Latent Representation for Co-speech Gesture Video Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2409.17674 |