An Evaluation of Large Pre-Trained Models for Gesture Recognition using Synthetic Videos
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929525491761152 |
|---|---|
| author | Reddy, Arun Shah, Ketul Rivera, Corban Paul, William De Melo, Celso M. Chellappa, Rama |
| author_facet | Reddy, Arun Shah, Ketul Rivera, Corban Paul, William De Melo, Celso M. Chellappa, Rama |
| contents | In this work, we explore the possibility of using synthetically generated data for video-based gesture recognition with large pre-trained models. We consider whether these models have sufficiently robust and expressive representation spaces to enable "training-free" classification. Specifically, we utilize various state-of-the-art video encoders to extract features for use in k-nearest neighbors classification, where the training data points are derived from synthetic videos only. We compare these results with another training-free approach -- zero-shot classification using text descriptions of each gesture. In our experiments with the RoCoG-v2 dataset, we find that using synthetic training videos yields significantly lower classification accuracy on real test videos compared to using a relatively small number of real training videos. We also observe that video backbones that were fine-tuned on classification tasks serve as superior feature extractors, and that the choice of fine-tuning data has a substantial impact on k-nearest neighbors performance. Lastly, we find that zero-shot text-based classification performs poorly on the gesture recognition task, as gestures are not easily described through natural language. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_02152 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | An Evaluation of Large Pre-Trained Models for Gesture Recognition using Synthetic Videos Reddy, Arun Shah, Ketul Rivera, Corban Paul, William De Melo, Celso M. Chellappa, Rama Computer Vision and Pattern Recognition In this work, we explore the possibility of using synthetically generated data for video-based gesture recognition with large pre-trained models. We consider whether these models have sufficiently robust and expressive representation spaces to enable "training-free" classification. Specifically, we utilize various state-of-the-art video encoders to extract features for use in k-nearest neighbors classification, where the training data points are derived from synthetic videos only. We compare these results with another training-free approach -- zero-shot classification using text descriptions of each gesture. In our experiments with the RoCoG-v2 dataset, we find that using synthetic training videos yields significantly lower classification accuracy on real test videos compared to using a relatively small number of real training videos. We also observe that video backbones that were fine-tuned on classification tasks serve as superior feature extractors, and that the choice of fine-tuning data has a substantial impact on k-nearest neighbors performance. Lastly, we find that zero-shot text-based classification performs poorly on the gesture recognition task, as gestures are not easily described through natural language. |
| title | An Evaluation of Large Pre-Trained Models for Gesture Recognition using Synthetic Videos |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.02152 |