Comparative Analysis of Modality Fusion Approaches for Audio-Visual Person Identification and Verification
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912101848580096 |
|---|---|
| author | Farhadipour, Aref Chapariniya, Masoumeh Vukovic, Teodora Dellwo, Volker |
| author_facet | Farhadipour, Aref Chapariniya, Masoumeh Vukovic, Teodora Dellwo, Volker |
| contents | Multimodal learning involves integrating information from various modalities to enhance learning and comprehension. We compare three modality fusion strategies in person identification and verification by processing two modalities: voice and face. In this paper, a one-dimensional convolutional neural network is employed for x-vector extraction from voice, while the pre-trained VGGFace2 network and transfer learning are utilized for face modality. In addition, gammatonegram is used as speech representation in engagement with the Darknet19 pre-trained network. The proposed systems are evaluated using the K-fold cross-validation technique on the 118 speakers of the test set of the VoxCeleb2 dataset. The comparative evaluations are done for single-modality and three proposed multimodal strategies in equal situations. Results demonstrate that the feature fusion strategy of gammatonegram and facial features achieves the highest performance, with an accuracy of 98.37% in the person identification task. However, concatenating facial features with the x-vector reaches 0.62% for EER in verification tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_00562 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Comparative Analysis of Modality Fusion Approaches for Audio-Visual Person Identification and Verification Farhadipour, Aref Chapariniya, Masoumeh Vukovic, Teodora Dellwo, Volker Audio and Speech Processing Computer Vision and Pattern Recognition Multimedia Sound Multimodal learning involves integrating information from various modalities to enhance learning and comprehension. We compare three modality fusion strategies in person identification and verification by processing two modalities: voice and face. In this paper, a one-dimensional convolutional neural network is employed for x-vector extraction from voice, while the pre-trained VGGFace2 network and transfer learning are utilized for face modality. In addition, gammatonegram is used as speech representation in engagement with the Darknet19 pre-trained network. The proposed systems are evaluated using the K-fold cross-validation technique on the 118 speakers of the test set of the VoxCeleb2 dataset. The comparative evaluations are done for single-modality and three proposed multimodal strategies in equal situations. Results demonstrate that the feature fusion strategy of gammatonegram and facial features achieves the highest performance, with an accuracy of 98.37% in the person identification task. However, concatenating facial features with the x-vector reaches 0.62% for EER in verification tasks. |
| title | Comparative Analysis of Modality Fusion Approaches for Audio-Visual Person Identification and Verification |
| topic | Audio and Speech Processing Computer Vision and Pattern Recognition Multimedia Sound |
| url | https://arxiv.org/abs/2409.00562 |