Towards AI-driven Sign Language Generation with Non-manual Markers
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929705824813056 |
|---|---|
| author | Zhang, Han Shalev-Arkushin, Rotem Baltatzis, Vasileios Gillis, Connor Laput, Gierad Kushalnagar, Raja Quandt, Lorna Findlater, Leah Bedri, Abdelkareem Lea, Colin |
| author_facet | Zhang, Han Shalev-Arkushin, Rotem Baltatzis, Vasileios Gillis, Connor Laput, Gierad Kushalnagar, Raja Quandt, Lorna Findlater, Leah Bedri, Abdelkareem Lea, Colin |
| contents | Sign languages are essential for the Deaf and Hard-of-Hearing (DHH) community. Sign language generation systems have the potential to support communication by translating from written languages, such as English, into signed videos. However, current systems often fail to meet user needs due to poor translation of grammatical structures, the absence of facial cues and body language, and insufficient visual and motion fidelity. We address these challenges by building on recent advances in LLMs and video generation models to translate English sentences into natural-looking AI ASL signers. The text component of our model extracts information for manual and non-manual components of ASL, which are used to synthesize skeletal pose sequences and corresponding video frames. Our findings from a user study with 30 DHH participants and thorough technical evaluations demonstrate significant progress and identify critical areas necessary to meet user needs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_05661 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards AI-driven Sign Language Generation with Non-manual Markers Zhang, Han Shalev-Arkushin, Rotem Baltatzis, Vasileios Gillis, Connor Laput, Gierad Kushalnagar, Raja Quandt, Lorna Findlater, Leah Bedri, Abdelkareem Lea, Colin Human-Computer Interaction Sign languages are essential for the Deaf and Hard-of-Hearing (DHH) community. Sign language generation systems have the potential to support communication by translating from written languages, such as English, into signed videos. However, current systems often fail to meet user needs due to poor translation of grammatical structures, the absence of facial cues and body language, and insufficient visual and motion fidelity. We address these challenges by building on recent advances in LLMs and video generation models to translate English sentences into natural-looking AI ASL signers. The text component of our model extracts information for manual and non-manual components of ASL, which are used to synthesize skeletal pose sequences and corresponding video frames. Our findings from a user study with 30 DHH participants and thorough technical evaluations demonstrate significant progress and identify critical areas necessary to meet user needs. |
| title | Towards AI-driven Sign Language Generation with Non-manual Markers |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2502.05661 |