FSboard: Over 3 million characters of ASL fingerspelling collected via smartphones
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909264383049728 |
|---|---|
| author | Georg, Manfred Tanzer, Garrett Hassan, Saad Shengelia, Maximus Uboweja, Esha Sepah, Sam Forbes, Sean Starner, Thad |
| author_facet | Georg, Manfred Tanzer, Garrett Hassan, Saad Shengelia, Maximus Uboweja, Esha Sepah, Sam Forbes, Sean Starner, Thad |
| contents | Progress in machine understanding of sign languages has been slow and hampered by limited data. In this paper, we present FSboard, an American Sign Language fingerspelling dataset situated in a mobile text entry use case, collected from 147 paid and consenting Deaf signers using Pixel 4A selfie cameras in a variety of environments. Fingerspelling recognition is an incomplete solution that is only one small part of sign language translation, but it could provide some immediate benefit to Deaf/Hard of Hearing signers as more broadly capable technology develops. At >3 million characters in length and >250 hours in duration, FSboard is the largest fingerspelling recognition dataset to date by a factor of >10x. As a simple baseline, we finetune 30 Hz MediaPipe Holistic landmark inputs into ByT5-Small and achieve 11.1% Character Error Rate (CER) on a test set with unique phrases and signers. This quality degrades gracefully when decreasing frame rate and excluding face/body landmarks: plausible optimizations to help models run on device in real time. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_15806 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FSboard: Over 3 million characters of ASL fingerspelling collected via smartphones Georg, Manfred Tanzer, Garrett Hassan, Saad Shengelia, Maximus Uboweja, Esha Sepah, Sam Forbes, Sean Starner, Thad Computer Vision and Pattern Recognition Computation and Language Progress in machine understanding of sign languages has been slow and hampered by limited data. In this paper, we present FSboard, an American Sign Language fingerspelling dataset situated in a mobile text entry use case, collected from 147 paid and consenting Deaf signers using Pixel 4A selfie cameras in a variety of environments. Fingerspelling recognition is an incomplete solution that is only one small part of sign language translation, but it could provide some immediate benefit to Deaf/Hard of Hearing signers as more broadly capable technology develops. At >3 million characters in length and >250 hours in duration, FSboard is the largest fingerspelling recognition dataset to date by a factor of >10x. As a simple baseline, we finetune 30 Hz MediaPipe Holistic landmark inputs into ByT5-Small and achieve 11.1% Character Error Rate (CER) on a test set with unique phrases and signers. This quality degrades gracefully when decreasing frame rate and excluding face/body landmarks: plausible optimizations to help models run on device in real time. |
| title | FSboard: Over 3 million characters of ASL fingerspelling collected via smartphones |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2407.15806 |