SignAvatar: Sign Language 3D Motion Reconstruction and Generation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916511791185920 |
|---|---|
| author | Dong, Lu Chaudhary, Lipisha Xu, Fei Wang, Xiao Lary, Mason Nwogu, Ifeoma |
| author_facet | Dong, Lu Chaudhary, Lipisha Xu, Fei Wang, Xiao Lary, Mason Nwogu, Ifeoma |
| contents | Achieving expressive 3D motion reconstruction and automatic generation for isolated sign words can be challenging, due to the lack of real-world 3D sign-word data, the complex nuances of signing motions, and the cross-modal understanding of sign language semantics. To address these challenges, we introduce SignAvatar, a framework capable of both word-level sign language reconstruction and generation. SignAvatar employs a transformer-based conditional variational autoencoder architecture, effectively establishing relationships across different semantic modalities. Additionally, this approach incorporates a curriculum learning strategy to enhance the model's robustness and generalization, resulting in more realistic motions. Furthermore, we contribute the ASL3DWord dataset, composed of 3D joint rotation data for the body, hands, and face, for unique sign words. We demonstrate the effectiveness of SignAvatar through extensive experiments, showcasing its superior reconstruction and automatic generation capabilities. The code and dataset are available on the project page. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_07974 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SignAvatar: Sign Language 3D Motion Reconstruction and Generation Dong, Lu Chaudhary, Lipisha Xu, Fei Wang, Xiao Lary, Mason Nwogu, Ifeoma Computer Vision and Pattern Recognition Achieving expressive 3D motion reconstruction and automatic generation for isolated sign words can be challenging, due to the lack of real-world 3D sign-word data, the complex nuances of signing motions, and the cross-modal understanding of sign language semantics. To address these challenges, we introduce SignAvatar, a framework capable of both word-level sign language reconstruction and generation. SignAvatar employs a transformer-based conditional variational autoencoder architecture, effectively establishing relationships across different semantic modalities. Additionally, this approach incorporates a curriculum learning strategy to enhance the model's robustness and generalization, resulting in more realistic motions. Furthermore, we contribute the ASL3DWord dataset, composed of 3D joint rotation data for the body, hands, and face, for unique sign words. We demonstrate the effectiveness of SignAvatar through extensive experiments, showcasing its superior reconstruction and automatic generation capabilities. The code and dataset are available on the project page. |
| title | SignAvatar: Sign Language 3D Motion Reconstruction and Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2405.07974 |