Bangla Sign Language Recognition With Multimodal Deep Learning Fusion
Fuente:
Wiley Open Access
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Artículo Open Access |
| Veröffentlicht: |
Wiley
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1867013712087351296 |
|---|---|
| author | Adib Hasan Mahmodul Hasan Jobayer Md Abdullah Al Mahmud Pias Tahmidul Alam Riasat Khan |
| author_facet | Adib Hasan Mahmodul Hasan Jobayer Md Abdullah Al Mahmud Pias Tahmidul Alam Riasat Khan Adib Hasan Mahmodul Hasan Jobayer Md Abdullah Al Mahmud Pias Tahmidul Alam Riasat Khan |
| collection | Wiley Open Access |
| contents | Bangla Sign Language Recognition With Multimodal Deep Learning Fusion Adib Hasan Mahmodul Hasan Jobayer Md Abdullah Al Mahmud Pias Tahmidul Alam Riasat Khan Engineering Reports ABSTRACTSign languages are conveyed through hand gestures and body language; different cultures and languages have unique sign language representations. It is difficult for the general population to interpret all these variations. A Bangla sign language recognition system can help mitigate this problem. In Bangladesh, approximately 13.7 million people are affected by hearing impairments, highlighting the importance of enhancing the Bangla sign language recognition system. In this study, a merged dataset consists of 200‐word classes and 7,000 videos performed by 16 signers. A unique aspect of this dataset is that it includes two modalities. Initially, we extracted frames and audio from the videos. We then employed two frameworks to capture body keypoints from the frames. Additionally, we contributed to expanding the dataset by creating a custom dataset that adds more samples per class. During preprocessing, we augmented the training data, improving our model's learning capacity and reducing overfitting. We developed a system leveraging various deep learning techniques, simultaneously working with two different modalities. We proposed a custom CNN‐LSTM model for pose estimation and integrated VGGish and 2D CNN techniques to process audio in our multimodal model. The CNN‐LSTM model attained an accuracy of 87.08% for pose estimation using the OpenPose framework. The applied ViT model achieved the best performance with 88.52% and 88.39% F1 score to predict body keypoints directly. The VGGish technique delivered the best result for the audio dataset, achieving 81.91% accuracy and 80.42% F1 score. Finally, the late fusion approach combining the ViT and VGGish networks employing the multimodal data achieved the best performance, achieving 94.71% accuracy and 94.52% F1 score. 10.1002/eng2.70139 http://creativecommons.org/licenses/by/4.0/ |
| doi_str_mv | 10.1002/eng2.70139 |
| format | Artículo Open Access |
| id | wiley_oa_10_1002_eng2_70139 |
| institution | Wiley Open Access |
| license_str_mv | http://creativecommons.org/licenses/by/4.0/ |
| publishDate | 2025 |
| publisher | Wiley |
| record_format | wiley_oa |
| spellingShingle | Bangla Sign Language Recognition With Multimodal Deep Learning Fusion Adib Hasan Mahmodul Hasan Jobayer Md Abdullah Al Mahmud Pias Tahmidul Alam Riasat Khan Engineering Reports Bangla Sign Language Recognition With Multimodal Deep Learning Fusion Adib Hasan Mahmodul Hasan Jobayer Md Abdullah Al Mahmud Pias Tahmidul Alam Riasat Khan Engineering Reports ABSTRACTSign languages are conveyed through hand gestures and body language; different cultures and languages have unique sign language representations. It is difficult for the general population to interpret all these variations. A Bangla sign language recognition system can help mitigate this problem. In Bangladesh, approximately 13.7 million people are affected by hearing impairments, highlighting the importance of enhancing the Bangla sign language recognition system. In this study, a merged dataset consists of 200‐word classes and 7,000 videos performed by 16 signers. A unique aspect of this dataset is that it includes two modalities. Initially, we extracted frames and audio from the videos. We then employed two frameworks to capture body keypoints from the frames. Additionally, we contributed to expanding the dataset by creating a custom dataset that adds more samples per class. During preprocessing, we augmented the training data, improving our model's learning capacity and reducing overfitting. We developed a system leveraging various deep learning techniques, simultaneously working with two different modalities. We proposed a custom CNN‐LSTM model for pose estimation and integrated VGGish and 2D CNN techniques to process audio in our multimodal model. The CNN‐LSTM model attained an accuracy of 87.08% for pose estimation using the OpenPose framework. The applied ViT model achieved the best performance with 88.52% and 88.39% F1 score to predict body keypoints directly. The VGGish technique delivered the best result for the audio dataset, achieving 81.91% accuracy and 80.42% F1 score. Finally, the late fusion approach combining the ViT and VGGish networks employing the multimodal data achieved the best performance, achieving 94.71% accuracy and 94.52% F1 score. 10.1002/eng2.70139 http://creativecommons.org/licenses/by/4.0/ |
| title | Bangla Sign Language Recognition With Multimodal Deep Learning Fusion |
| topic | Engineering Reports |
| url | https://onlinelibrary.wiley.com/doi/10.1002/eng2.70139 |