Improving the Performance of Unimodal Dynamic Hand-Gesture Recognition with Multimodal Training
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2018
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915543858020352 |
|---|---|
| author | Abavisani, Mahdi Joze, Hamid Reza Vaezi Patel, Vishal M. |
| author_facet | Abavisani, Mahdi Joze, Hamid Reza Vaezi Patel, Vishal M. |
| contents | We present an efficient approach for leveraging the knowledge from multiple modalities in training unimodal 3D convolutional neural networks (3D-CNNs) for the task of dynamic hand gesture recognition. Instead of explicitly combining multimodal information, which is commonplace in many state-of-the-art methods, we propose a different framework in which we embed the knowledge of multiple modalities in individual networks so that each unimodal network can achieve an improved performance. In particular, we dedicate separate networks per available modality and enforce them to collaborate and learn to develop networks with common semantics and better representations. We introduce a "spatiotemporal semantic alignment" loss (SSA) to align the content of the features from different networks. In addition, we regularize this loss with our proposed "focal regularization parameter" to avoid negative knowledge transfer. Experimental results show that our framework improves the test time recognition accuracy of unimodal networks, and provides the state-of-the-art performance on various dynamic hand gesture recognition datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_1812_06145 |
| institution | arXiv |
| publishDate | 2018 |
| record_format | arxiv |
| spellingShingle | Improving the Performance of Unimodal Dynamic Hand-Gesture Recognition with Multimodal Training Abavisani, Mahdi Joze, Hamid Reza Vaezi Patel, Vishal M. Computer Vision and Pattern Recognition Artificial Intelligence Human-Computer Interaction Machine Learning 68T45, 62H30 I.5.3; I.2.10 We present an efficient approach for leveraging the knowledge from multiple modalities in training unimodal 3D convolutional neural networks (3D-CNNs) for the task of dynamic hand gesture recognition. Instead of explicitly combining multimodal information, which is commonplace in many state-of-the-art methods, we propose a different framework in which we embed the knowledge of multiple modalities in individual networks so that each unimodal network can achieve an improved performance. In particular, we dedicate separate networks per available modality and enforce them to collaborate and learn to develop networks with common semantics and better representations. We introduce a "spatiotemporal semantic alignment" loss (SSA) to align the content of the features from different networks. In addition, we regularize this loss with our proposed "focal regularization parameter" to avoid negative knowledge transfer. Experimental results show that our framework improves the test time recognition accuracy of unimodal networks, and provides the state-of-the-art performance on various dynamic hand gesture recognition datasets. |
| title | Improving the Performance of Unimodal Dynamic Hand-Gesture Recognition with Multimodal Training |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Human-Computer Interaction Machine Learning 68T45, 62H30 I.5.3; I.2.10 |
| url | https://arxiv.org/abs/1812.06145 |