Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2309.07115 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914955442257920 |
|---|---|
| author | Selvakumar, Anith Fashandi, Homa |
| author_facet | Selvakumar, Anith Fashandi, Homa |
| contents | Distance Metric Learning (DML) has typically dominated the audio-visual speaker verification problem space, owing to strong performance in new and unseen classes. In our work, we explored multitask learning techniques to further enhance DML, and show that an auxiliary task with even weak labels can increase the quality of the learned speaker representation without increasing model complexity during inference. We also extend the Generalized End-to-End Loss (GE2E) to multimodal inputs and demonstrate that it can achieve competitive performance in an audio-visual space. Finally, we introduce AV-Mixup, a multimodal augmentation technique during training time that has shown to reduce speaker overfit. Our network achieves state of the art performance for speaker verification, reporting 0.244%, 0.252%, 0.441% Equal Error Rate (EER) on the VoxCeleb1-O/E/H test sets, which is to our knowledge, the best published results on VoxCeleb1-E and VoxCeleb1-H. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_07115 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Getting More for Less: Using Weak Labels and AV-Mixup for Robust Audio-Visual Speaker Verification Selvakumar, Anith Fashandi, Homa Sound Computer Vision and Pattern Recognition Machine Learning Multimedia Audio and Speech Processing Distance Metric Learning (DML) has typically dominated the audio-visual speaker verification problem space, owing to strong performance in new and unseen classes. In our work, we explored multitask learning techniques to further enhance DML, and show that an auxiliary task with even weak labels can increase the quality of the learned speaker representation without increasing model complexity during inference. We also extend the Generalized End-to-End Loss (GE2E) to multimodal inputs and demonstrate that it can achieve competitive performance in an audio-visual space. Finally, we introduce AV-Mixup, a multimodal augmentation technique during training time that has shown to reduce speaker overfit. Our network achieves state of the art performance for speaker verification, reporting 0.244%, 0.252%, 0.441% Equal Error Rate (EER) on the VoxCeleb1-O/E/H test sets, which is to our knowledge, the best published results on VoxCeleb1-E and VoxCeleb1-H. |
| title | Getting More for Less: Using Weak Labels and AV-Mixup for Robust Audio-Visual Speaker Verification |
| topic | Sound Computer Vision and Pattern Recognition Machine Learning Multimedia Audio and Speech Processing |
| url | https://arxiv.org/abs/2309.07115 |