Integrating Audio Narrations to Strengthen Domain Generalization in Multimodal First-Person Action Recognition
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909316742643712 |
|---|---|
| author | Gungor, Cagri Kovashka, Adriana |
| author_facet | Gungor, Cagri Kovashka, Adriana |
| contents | First-person activity recognition is rapidly growing due to the widespread use of wearable cameras but faces challenges from domain shifts across different environments, such as varying objects or background scenes. We propose a multimodal framework that improves domain generalization by integrating motion, audio, and appearance features. Key contributions include analyzing the resilience of audio and motion features to domain shifts, using audio narrations for enhanced audio-text alignment, and applying consistency ratings between audio and visual narrations to optimize the impact of audio in recognition during training. Our approach achieves state-of-the-art performance on the ARGO1M dataset, effectively generalizing across unseen scenarios and locations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_09611 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Integrating Audio Narrations to Strengthen Domain Generalization in Multimodal First-Person Action Recognition Gungor, Cagri Kovashka, Adriana Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Sound Audio and Speech Processing First-person activity recognition is rapidly growing due to the widespread use of wearable cameras but faces challenges from domain shifts across different environments, such as varying objects or background scenes. We propose a multimodal framework that improves domain generalization by integrating motion, audio, and appearance features. Key contributions include analyzing the resilience of audio and motion features to domain shifts, using audio narrations for enhanced audio-text alignment, and applying consistency ratings between audio and visual narrations to optimize the impact of audio in recognition during training. Our approach achieves state-of-the-art performance on the ARGO1M dataset, effectively generalizing across unseen scenarios and locations. |
| title | Integrating Audio Narrations to Strengthen Domain Generalization in Multimodal First-Person Action Recognition |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2409.09611 |