Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2308.16082 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918004779909120 |
|---|---|
| author | Fang, Sen Sui, Chunyu Zhou, Yanghao Zhang, Xuedong Zhong, Hongbin Tian, Yapeng Chen, Chen |
| author_facet | Fang, Sen Sui, Chunyu Zhou, Yanghao Zhang, Xuedong Zhong, Hongbin Tian, Yapeng Chen, Chen |
| contents | In this paper, we propose a dual-condition diffusion pre-training model named SignDiff that can generate human sign language speakers from a skeleton pose. SignDiff has a novel Frame Reinforcement Network called FR-Net, similar to dense human pose estimation work, which enhances the correspondence between text lexical symbols and sign language dense pose frames, reduces the occurrence of multiple fingers in the diffusion model. In addition, we propose a new method for American Sign Language Production (ASLP), which can generate ASL skeletal pose videos from text input, integrating two new improved modules and a new loss function to improve the accuracy and quality of sign language skeletal posture and enhance the ability of the model to train on large-scale data. We propose the first baseline for ASL production and report the scores of 17.19 and 12.85 on BLEU-4 on the How2Sign dev/test sets. We evaluated our model on the previous mainstream dataset PHOENIX14T, and the experiments achieved the SOTA results. In addition, our image quality far exceeds all previous results by 10 percentage points in terms of SSIM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2308_16082 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | SignDiff: Diffusion Model for American Sign Language Production Fang, Sen Sui, Chunyu Zhou, Yanghao Zhang, Xuedong Zhong, Hongbin Tian, Yapeng Chen, Chen Computer Vision and Pattern Recognition In this paper, we propose a dual-condition diffusion pre-training model named SignDiff that can generate human sign language speakers from a skeleton pose. SignDiff has a novel Frame Reinforcement Network called FR-Net, similar to dense human pose estimation work, which enhances the correspondence between text lexical symbols and sign language dense pose frames, reduces the occurrence of multiple fingers in the diffusion model. In addition, we propose a new method for American Sign Language Production (ASLP), which can generate ASL skeletal pose videos from text input, integrating two new improved modules and a new loss function to improve the accuracy and quality of sign language skeletal posture and enhance the ability of the model to train on large-scale data. We propose the first baseline for ASL production and report the scores of 17.19 and 12.85 on BLEU-4 on the How2Sign dev/test sets. We evaluated our model on the previous mainstream dataset PHOENIX14T, and the experiments achieved the SOTA results. In addition, our image quality far exceeds all previous results by 10 percentage points in terms of SSIM. |
| title | SignDiff: Diffusion Model for American Sign Language Production |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2308.16082 |