MS2SL: Multimodal Spoken Data-Driven Continuous Sign Language Production
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911959131095040 |
|---|---|
| author | Ma, Jian Wang, Wenguan Yang, Yi Zheng, Feng |
| author_facet | Ma, Jian Wang, Wenguan Yang, Yi Zheng, Feng |
| contents | Sign language understanding has made significant strides; however, there is still no viable solution for generating sign sequences directly from entire spoken content, e.g., text or speech. In this paper, we propose a unified framework for continuous sign language production, easing communication between sign and non-sign language users. In particular, a sequence diffusion model, utilizing embeddings extracted from text or speech, is crafted to generate sign predictions step by step. Moreover, by creating a joint embedding space for text, audio, and sign, we bind these modalities and leverage the semantic consistency among them to provide informative feedback for the model training. This embedding-consistency learning strategy minimizes the reliance on sign triplets and ensures continuous model refinement, even with a missing audio modality. Experiments on How2Sign and PHOENIX14T datasets demonstrate that our model achieves competitive performance in sign language production. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_12842 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MS2SL: Multimodal Spoken Data-Driven Continuous Sign Language Production Ma, Jian Wang, Wenguan Yang, Yi Zheng, Feng Computation and Language Artificial Intelligence Sign language understanding has made significant strides; however, there is still no viable solution for generating sign sequences directly from entire spoken content, e.g., text or speech. In this paper, we propose a unified framework for continuous sign language production, easing communication between sign and non-sign language users. In particular, a sequence diffusion model, utilizing embeddings extracted from text or speech, is crafted to generate sign predictions step by step. Moreover, by creating a joint embedding space for text, audio, and sign, we bind these modalities and leverage the semantic consistency among them to provide informative feedback for the model training. This embedding-consistency learning strategy minimizes the reliance on sign triplets and ensures continuous model refinement, even with a missing audio modality. Experiments on How2Sign and PHOENIX14T datasets demonstrate that our model achieves competitive performance in sign language production. |
| title | MS2SL: Multimodal Spoken Data-Driven Continuous Sign Language Production |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2407.12842 |