MDT-A2G: Exploring Masked Diffusion Transformers for Co-Speech Gesture Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909280646463488 |
|---|---|
| author | Mao, Xiaofeng Jiang, Zhengkai Wang, Qilin Fu, Chencan Zhang, Jiangning Wu, Jiafu Wang, Yabiao Wang, Chengjie Li, Wei Chi, Mingmin |
| author_facet | Mao, Xiaofeng Jiang, Zhengkai Wang, Qilin Fu, Chencan Zhang, Jiangning Wu, Jiafu Wang, Yabiao Wang, Chengjie Li, Wei Chi, Mingmin |
| contents | Recent advancements in the field of Diffusion Transformers have substantially improved the generation of high-quality 2D images, 3D videos, and 3D shapes. However, the effectiveness of the Transformer architecture in the domain of co-speech gesture generation remains relatively unexplored, as prior methodologies have predominantly employed the Convolutional Neural Network (CNNs) or simple a few transformer layers. In an attempt to bridge this research gap, we introduce a novel Masked Diffusion Transformer for co-speech gesture generation, referred to as MDT-A2G, which directly implements the denoising process on gesture sequences. To enhance the contextual reasoning capability of temporally aligned speech-driven gestures, we incorporate a novel Masked Diffusion Transformer. This model employs a mask modeling scheme specifically designed to strengthen temporal relation learning among sequence gestures, thereby expediting the learning process and leading to coherent and realistic motions. Apart from audio, Our MDT-A2G model also integrates multi-modal information, encompassing text, emotion, and identity. Furthermore, we propose an efficient inference strategy that diminishes the denoising computation by leveraging previously calculated results, thereby achieving a speedup with negligible performance degradation. Experimental results demonstrate that MDT-A2G excels in gesture generation, boasting a learning speed that is over 6$\times$ faster than traditional diffusion transformers and an inference speed that is 5.7$\times$ than the standard diffusion model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_03312 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MDT-A2G: Exploring Masked Diffusion Transformers for Co-Speech Gesture Generation Mao, Xiaofeng Jiang, Zhengkai Wang, Qilin Fu, Chencan Zhang, Jiangning Wu, Jiafu Wang, Yabiao Wang, Chengjie Li, Wei Chi, Mingmin Computer Vision and Pattern Recognition Recent advancements in the field of Diffusion Transformers have substantially improved the generation of high-quality 2D images, 3D videos, and 3D shapes. However, the effectiveness of the Transformer architecture in the domain of co-speech gesture generation remains relatively unexplored, as prior methodologies have predominantly employed the Convolutional Neural Network (CNNs) or simple a few transformer layers. In an attempt to bridge this research gap, we introduce a novel Masked Diffusion Transformer for co-speech gesture generation, referred to as MDT-A2G, which directly implements the denoising process on gesture sequences. To enhance the contextual reasoning capability of temporally aligned speech-driven gestures, we incorporate a novel Masked Diffusion Transformer. This model employs a mask modeling scheme specifically designed to strengthen temporal relation learning among sequence gestures, thereby expediting the learning process and leading to coherent and realistic motions. Apart from audio, Our MDT-A2G model also integrates multi-modal information, encompassing text, emotion, and identity. Furthermore, we propose an efficient inference strategy that diminishes the denoising computation by leveraging previously calculated results, thereby achieving a speedup with negligible performance degradation. Experimental results demonstrate that MDT-A2G excels in gesture generation, boasting a learning speed that is over 6$\times$ faster than traditional diffusion transformers and an inference speed that is 5.7$\times$ than the standard diffusion model. |
| title | MDT-A2G: Exploring Masked Diffusion Transformers for Co-Speech Gesture Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.03312 |