ASL Champ!: A Virtual Reality Game with Deep-Learning Driven Sign Recognition
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909058342060032 |
|---|---|
| author | Alam, Md Shahinur Lamberton, Jason Wang, Jianye Leannah, Carly Miller, Sarah Palagano, Joseph de Bastion, Myles Smith, Heather L. Malzkuhn, Melissa Quandt, Lorna C. |
| author_facet | Alam, Md Shahinur Lamberton, Jason Wang, Jianye Leannah, Carly Miller, Sarah Palagano, Joseph de Bastion, Myles Smith, Heather L. Malzkuhn, Melissa Quandt, Lorna C. |
| contents | We developed an American Sign Language (ASL) learning platform in a Virtual Reality (VR) environment to facilitate immersive interaction and real-time feedback for ASL learners. We describe the first game to use an interactive teaching style in which users learn from a fluent signing avatar and the first implementation of ASL sign recognition using deep learning within the VR environment. Advanced motion-capture technology powers an expressive ASL teaching avatar within an immersive three-dimensional environment. The teacher demonstrates an ASL sign for an object, prompting the user to copy the sign. Upon the user's signing, a third-party plugin executes the sign recognition process alongside a deep learning model. Depending on the accuracy of a user's sign production, the avatar repeats the sign or introduces a new one. We gathered a 3D VR ASL dataset from fifteen diverse participants to power the sign recognition model. The proposed deep learning model's training, validation, and test accuracy are 90.12%, 89.37%, and 86.66%, respectively. The functional prototype can teach sign language vocabulary and be successfully adapted as an interactive ASL learning platform in VR. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_00289 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | ASL Champ!: A Virtual Reality Game with Deep-Learning Driven Sign Recognition Alam, Md Shahinur Lamberton, Jason Wang, Jianye Leannah, Carly Miller, Sarah Palagano, Joseph de Bastion, Myles Smith, Heather L. Malzkuhn, Melissa Quandt, Lorna C. Human-Computer Interaction We developed an American Sign Language (ASL) learning platform in a Virtual Reality (VR) environment to facilitate immersive interaction and real-time feedback for ASL learners. We describe the first game to use an interactive teaching style in which users learn from a fluent signing avatar and the first implementation of ASL sign recognition using deep learning within the VR environment. Advanced motion-capture technology powers an expressive ASL teaching avatar within an immersive three-dimensional environment. The teacher demonstrates an ASL sign for an object, prompting the user to copy the sign. Upon the user's signing, a third-party plugin executes the sign recognition process alongside a deep learning model. Depending on the accuracy of a user's sign production, the avatar repeats the sign or introduces a new one. We gathered a 3D VR ASL dataset from fifteen diverse participants to power the sign recognition model. The proposed deep learning model's training, validation, and test accuracy are 90.12%, 89.37%, and 86.66%, respectively. The functional prototype can teach sign language vocabulary and be successfully adapted as an interactive ASL learning platform in VR. |
| title | ASL Champ!: A Virtual Reality Game with Deep-Learning Driven Sign Recognition |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2401.00289 |